bazel: migrate curl & openssl to dd_cc_packaged & dd_collect_dependencies#49403
bazel: migrate curl & openssl to dd_cc_packaged & dd_collect_dependencies#49403chouquette wants to merge 15 commits intomainfrom
Conversation
adc2142 to
a43e57d
Compare
509ee09 to
13e245b
Compare
a43e57d to
aeb3d26
Compare
aeb3d26 to
c546639
Compare
532c884 to
cb43834
Compare
c546639 to
8f3c11a
Compare
This way we don't need to patch the binary afterward, which is more robust and much easier to reason about
When using foreign_cc_shared_wrapper on the same input but to extract 2 different libraries, we end up with the same label being exported twice which causes some rather obscure build error down the line: ERROR: /home/hugo.beauzee/.cache/bazel/_bazel_hugo.beauzee/95605d1a73395e0c041bb302da387b0e/external/+_repo_rules+curl/BUILD.bazel:222:18: in cc_shared_library rule @@+_repo_rules+curl//:curl: Traceback (most recent call last): File "/virtual_builtins_bzl/common/cc/cc_shared_library.bzl", line 654, column 60, in _cc_shared_library_impl File "/virtual_builtins_bzl/common/cc/cc_shared_library.bzl", line 251, column 33, in _build_exports_map_from_only_dynamic_deps Error: unsupported binary operation: string + Label
This is useful for dependencies such as openssl which ships 2 libraries from the same configure_make label
This allows us to use the wrapper object as the input for other rules such as rewrite_rpath
When the input comes from a label we can't easily adapt to the expected convention (foo for a label that points at libfoo.so) we need a way to ensure we don't generate a non-sensical symlink filename
Instead, add curl to the leaf dependencies entrypoint
This allows a dependency to also package its executables and have them rpath patched automatically
cb43834 to
a27f904
Compare
Files inventory check summaryFile checks results against ancestor bdc7100e: Results for datadog-agent_7.80.0~devel.git.89.74105c4.pipeline.108818066-1_amd64.deb:Detected file changes:
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: bdc7100 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +2.36 | [-0.61, +5.32] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +2.90 | [+2.68, +3.13] | 1 | Logs |
| ➖ | docker_containers_cpu | % cpu utilization | +2.36 | [-0.61, +5.32] | 1 | Logs |
| ➖ | ddot_logs | memory utilization | +0.47 | [+0.41, +0.54] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.27 | [+0.20, +0.33] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.20 | [+0.15, +0.25] | 1 | Logs bounds checks dashboard |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | +0.13 | [+0.06, +0.19] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.12 | [-0.29, +0.52] | 1 | Logs |
| ➖ | ddot_metrics | memory utilization | +0.10 | [-0.09, +0.29] | 1 | Logs |
| ➖ | docker_containers_memory | memory utilization | +0.08 | [+0.01, +0.16] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | +0.06 | [-0.07, +0.20] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_v3 | ingress throughput | +0.03 | [-0.20, +0.25] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.01 | [-0.11, +0.12] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | +0.00 | [-0.42, +0.43] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.00 | [-0.54, +0.55] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.04 | [-0.24, +0.16] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.12 | [-0.16, -0.09] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics_sum_delta | memory utilization | -0.13 | [-0.30, +0.05] | 1 | Logs |
| ➖ | otlp_ingest_metrics | memory utilization | -0.18 | [-0.33, -0.03] | 1 | Logs |
| ➖ | quality_gate_metrics_logs | memory utilization | -0.22 | [-0.45, +0.02] | 1 | Logs bounds checks dashboard |
| ➖ | otlp_ingest_logs | memory utilization | -0.23 | [-0.33, -0.12] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulativetodelta_exporter | memory utilization | -0.27 | [-0.49, -0.05] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulative | memory utilization | -0.32 | [-0.47, -0.16] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | -2.80 | [-4.41, -1.19] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | 684 ≥ 26 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | 277.78MiB ≤ 370MiB | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | 649 ≥ 26 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | 0.19GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_0ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | 0.24GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_1000ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | 0.20GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_100ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | 0.22GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_500ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 176.09MiB ≤ 181MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 499.59MiB ≤ 550MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 211.37MiB ≤ 220MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 345.66 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 426.05MiB ≤ 475MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
What does this PR do?
Continue the migration toward installing dependencies with dd_collect_dependencies
This PR mostly handles OpenSSL, using cURL as the entry point that depends on it.
Most of the changes are actually related to our rules_foreign_cc wrapper, so that it integrates nicely with dd_cc_packaged
Motivation
Migrating away from omnibus.
Describe how you validated your changes
Local
bazel test //...and CIAdditional Notes
Based on top of #49386(merged)Based on top of #49500
The rules_pkg change is opened upstream at bazelbuild/rules_pkg#1053