Share
Embed
きっと明日から役立つeBPFのしくみ
第15回カーネル/VM探検隊での発表. eBPFのローダの仕組みやVerifierの仕組みなどについて1つのeBPFプログラムを題材に掘り下げます.
Yutaro Hayakawa
July 20, 2019
Transcript
58 slides
-
ࣗݾհ • ໊લૣါଠ࿕ ͔Θ Ώ͏ͨΖ͏ • ॴଐ-*/&גࣜձࣾ ωοτϫʔΫ։ൃνʔϜ • 5XJUUFS!:VUBSP)BZBLBXB
• (4P$ ͰF#1'Λ'SFF#4%Ҡ২͢ΔϓϩδΣΫτ Λ͍ͬͯͨ ࠓ͍ͬͯΔ • ͖ͳ • ߴύέοτॲཧ /FUNBQ 9%1 • -JOVY 'SFF#4%ͷωοτϫʔΫελοΫपΓ /FUGJMUFS 1' FUD • ωοτϫʔΫϓϩάϥϚϏϦςΟతͳ 1 FUD • F#1' ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 1 Twitterのアイコン -
ࠓͷ • ࠷ۙ-JOVYք۾ͰF#1'͕Ξπ͍ • ʮF#1'Λͬͨͳʹ͔ʯʹ͍͍ͭͯͩͿใ͕ग़ͯ͘ΔΑ͏ʹͳͬͨ • ʮF#1'ͦͷͷʯʹ͍ͭͯใ͕গͳ͍ • ී௨ʹF#1'Λ͍ͬͯΔͱϒϥοΫϘοΫεʹͳΓ͕ͪͳͱ͜ΖΛݟ͍ͯ͘ •
جຊతʹϚχΞοΫͳͰ͢ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 2 -
F#1'ͱʁ • &YUFOEFE#FSLFMFZ1BDLFU'JMUFS • -JOVYΧʔωϧͷ֦ுػೳΛॻͨ͘Ίͷ%4- -JOVYd • ΞηϯϒϦͬΆ͍ • UDQEVNQͱ͔XJSFTIBSLͰΘΕ͍ͯΔ#1'ͷ֦ு
͕ͩผ • -JOVYΧʔωϧͷதʹΠϯλʔϓϦλͱ+*5ίϯύΠϥ͕ೖ͍ͬͯΔ • 1BDLFU'JMUFSͱݴ͍ͬͯΔ͕ ࣮ࡍʹͬͱ͍ΖΜͳͱ͜ΖͰΘΕ͍ͯΔ • γεςϜίʔϧͷϑΟϧλϦϯά 4FDDPNQ • μΠφϛοΫτϨʔγϯά LQSPCF QFSG • ύέοτॲཧ 9%1 5$ 4PDLFU'JMUFS • FUD ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 5 -
ݴޠ༷ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 6 • 3*4$ͬΆ͍*4" • CJUͷݻఆ໋ྩ • ݻఆͷελοΫ
• CJUͷϨδελຊ ൚༻SdS ϑϨʔϜϙΠϯλS • "-6 +.1 -%45ܥ໋ྩ͕Ұ௨Γ • "UPNJD໋ྩ #1'༝དྷͷύέοτόοϑΝΛૢ࡞͢Δ໋ྩ • $BMM໋ྩ͕͋Δ IFMQFSGVODUJPOTͱݺΕΔΧʔωϧͷ͕ؔݺΔ • ॻ͘ͷ͠ΜͲ͍ͷͰ$Ͱॻ͍ͯίϯύΠϧ͢Δ • --7.ͷόοΫΤϯυ͕͋Δ struct bpf_insn { __u8 opcode; __u8 dst_reg:4 __u8 src_reg:4 __s16 off; __s32 imm }; -
$ͷݺͼग़͠نଇʹؔ͢Δ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 7 eBPF x86-64 分類 R0 RAX Return
value from function R1 RDI 第1引数 R2 RSI 第2引数 R3 RDX 第3引数 R4 RCX 第4引数 R5 R8 第5引数 R6 RBX Callee saved R7 R13 Callee saved R8 R14 Callee saved R9 R15 Callee saved R10 RBP Frame pointer (read only) $ͷݺͼग़͠نଇ͕+*5ͨ͠ͱ͖ʹYͱ"3.ͱίϯύνʹͳΔΑ͏ʹ࡞ΒΕ͍ͯΔ ''*ͷΦʔόʔϔου͕ͱͯগͳ͍ -
.BQ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 8 • F#1'ͱϢʔβεϖʔε͔Β͑Δ,74 • "SSBZ )BTI 5SJF
2VFVF 4UBDL FUD • MPPLVQVQEBUFEFMFUFFORVFVFEFRVFVFQFFL • F#1'͔ΒIFMQFSGVODUJPO • Ϣʔβεϖʔε͔ΒCQG • Ϣʔεέʔε • Χϯλ • ϧʔςΟϯάςʔϒϧ Map User App eBPF Program A eBPF Program B User Kernel -
CQG ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 9 • int bpf(int cmd, union
bpf_attr *attr, unsigned int size); • F#1'ͷΈશൠΛѻ͏γεςϜίʔϧ • όΠτίʔυͷϩʔυ .BQͷ࡞ FUD enum bpf_cmd { BPF_MAP_CREATE, BPF_MAP_LOOKUP_ELEM, BPF_MAP_UPDATE_ELEM, BPF_MAP_DELETE_ELEM, BPF_MAP_GET_NEXT_KEY, BPF_PROG_LOAD, ... }; union bpf_attr { struct { /* BPF_MAP_CREATE */ __u32 map_type; __u32 key_size; ... }; struct { /* BPF_PROG_LOAD */ __u32 prog_type; __u32 insn_cnt; __aligned_u64 insns; ... }; -
7FSJGJFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 10 • F#1'όΠτίʔυࣗମʹηΩϡϦςΟతͳྀͳ͍ • Χʔωϧͷ7FSJGJFS͕҆શʹ࣮ߦͰ͖Δ͔੩తʹݕূ͢Δ • ϧʔϓجຊతʹېࢭ
• +.1໋ྩͰมͳͱ͜Ζʹ΅͏ͱ͍ͯ͠ͳ͍͔ʁ • -%45Ͱมͳͱ͜Ζ͔ΒಡΈॻ͖͠Α͏ͱ͍ͯ͠ͳ͍͔ʁ • FUD FUD FUD • νΣοΫΛ௨Βͳ͚Εϩʔυ͢ΒͰ͖ͳ͍ • ຊʹ͜ΕͰେৎͳͷ͔ʁʁ • 4QFDUSF Byte Code User App eBPF Verifier User Kernel bpf (2) fd EPERM Pass! Fail! -
ຊͷओ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 12 __attribute__((section("maps"))) struct bpf_elf_map pkt_counter = {
.type = BPF_MAP_TYPE_ARRAY, .size_key = sizeof(uint32_t), .size_value = sizeof(uint32_t), .max_elem = 1 }; __attribute__((section("xdp"))) int xdp_program(struct xdp_md *ctx) { uint32_t *count, index = 0; count = bpf_map_lookup_elem(&pkt_counter, &index); if (count == NULL) { return XDP_DROP; } __sync_fetch_and_add(count, 1); return XDP_PASS; } ड৴ύέοτͷΛ͑Δ͚ͩͷ F#1' 9%1 ͷϓϩάϥϜ ͜ͷϓϩάϥϜ͕ʮͳΜͰ͏·͘ಈ͘ͷ ͔ʁʯҙ֎ͱਂ۷Γ͠ͳ͍ͱΘ͔Βͳ ͍ -
ԿΛ͍ͯ͠Δ͔ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 13 ཁૉͷ"SSBZ.BQΛએݴ .BQͷݕࡧ WBMVFͷϙΠϯλ Λऔಘ WBMVFΛJOQMBDFͰߋ৽ __attribute__((section("maps")))
struct bpf_elf_map pkt_counter = { .type = BPF_MAP_TYPE_ARRAY, .size_key = sizeof(uint32_t), .size_value = sizeof(uint32_t), .max_elem = 1 }; __attribute__((section("xdp"))) int xdp_program(struct xdp_md *ctx) { uint32_t *count, index = 0; count = bpf_map_lookup_elem(&pkt_counter, &index); if (count == NULL) { return XDP_DROP; } __sync_fetch_and_add(count, 1); return XDP_PASS; } -
F#1'ͷ$ϓϩάϥϜ͕࣮ߦ͞ΕΔ·Ͱ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 15 NIC driver bpf(2) bpftool demo.c clang
Verifier ここで動く! __attribute__((section("maps"))) struct bpf_elf_map pkt_counter = { .type = BPF_MAP_TYPE_ARRAY, .size_key = sizeof(uint32_t), .size_value = sizeof(uint32_t), .max_elem = 1 }; __attribute__((section("xdp"))) int xdp_program(struct xdp_md *ctx) { uint32_t *count, index = 0; count = bpf_map_lookup_elem(&pkt_counter, &index); if (count == NULL) { return XDP_DROP; } __sync_fetch_and_add(count, 1); return XDP_PASS; } -
F#1'ͷ$ϓϩάϥϜ͕࣮ߦ͞ΕΔ·Ͱ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 16 NIC driver bpf(2) bpftool demo.c clang
Verifier ここで動く! CQGUPPM ϩʔμ ͷ͘͠Έ 7FSJGJFSͷ͘͠Έ -
F#1'ͷ$ϓϩάϥϜ͕࣮ߦ͞ΕΔ·Ͱ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 17 NIC driver bpf(2) bpftool demo.c clang
Verifier ここで動く! CQGUPPM ϩʔμ ͷ͘͠Έ 7FSJGJFSͷ͘͠Έ -
F#1' -PBEFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 18 • F#1'ϓϩάϥϜΛΧʔωϧʹϩʔυ͢ΔϢʔβεϖʔεͷϓϩάϥϜ • ࣮ͭͰͳ͍ CQGUPPM
JQSPVUF CDD FUD • ͦΕͧΕඍົʹΈ͕ҧ͏ • ओͳࣄ &-'ϑΝΠϧ͔ΒϩʔυʹඞཁͳใΛऔΓग़͢ όΠτίʔυΛม͢Δ CQG Λͬͯϩʔυ͢Δ -
F#1' -PBEFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 19 • F#1'ϓϩάϥϜΛΧʔωϧʹϩʔυ͢ΔϢʔβεϖʔεͷϓϩάϥϜ • ࣮ͭͰͳ͍ CQGUPPM
JQSPVUF CDD FUD • ͦΕͧΕඍົʹΈ͕ҧ͏ • ओͳࣄ &-'ϑΝΠϧ͔ΒϩʔυʹඞཁͳใΛऔΓग़͢ όΠτίʔυΛม͢Δ CQG Λͬͯϩʔυ͢Δ -
CQG ͰόΠτίʔυΛϩʔυ͢Δʹ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 20 • #1'@130(@-0"%ίϚϯυΛ͏ • ͜ͷγεςϜίʔϧͷҾΛ&-'ϑΝΠϧ͔Β͔͖ूΊΔ
struct { __u32 prog_type; // プログラムの”タイプ” (XDP, kprobe,etc…) __u32 insn_cnt; // 命令の数 __aligned_u64 insns; // eBPFバイトコードへのポインタ __aligned_u64 license; // ライセンス⽂字列へのポインタ __u32 log_level; // Verifierのログレベル __u32 log_size; // ログバッファのサイズ __aligned_u64 log_buf; // ログバッファへのポインタ __u32 kern_version; // カーネルのバージョン __u32 prog_flags; // フラグ char prog_name[BPF_OBJ_NAME_LEN]; // 名前 __u32 prog_ifindex; // (XDPオフローディングに使われる) }; -
&-'ͷͲ͔͜Βͱͬͯ͘Δ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 21 struct { __u32 prog_type; // バイトコードが置かれているセクションの名前で決める
__u32 insn_cnt; // ELFのメタデータ __aligned_u64 insns; // PROGBITS Typeのついた最初のセクション __aligned_u64 license; // licenseセクションから取ってくる __u32 log_level; __u32 log_size; __aligned_u64 log_buf; __u32 kern_version; // versionセクションから取ってくる __u32 prog_flags; char prog_name[BPF_OBJ_NAME_LEN]; // 関数のシンボル名 __u32 prog_ifindex; }; • CQGUPPMͷ߹ ଞͷπʔϧࣅͨΑ͏ͳײ͡ -
&-'ηΫγϣϯͷ໋໊نଇ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 22 • CQGUPPMͷ߹ • ϓϩάϥϜͷλΠϓ<λΠϓݻ༗ͷύϥϝʔλ> • ྫ
YEQ LQSPCFDPOUFYU@TXJUDI • όΠτίʔυΛஔ͍͓ͯ͘ηΫγϣϯ • NBQT.BQͷใΛஔ͍͓ͯ͘ηΫγϣϯ • WFSTJPO-JOVYΧʔωϧͷόʔδϣϯίʔυΛஔ͍͓ͯ͘ηΫγϣϯ • MJDFOTFϥΠηϯεͷจࣈྻΛஔ͘ηΫγϣϯ -
F#1' -PBEFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 23 • F#1'ϓϩάϥϜΛΧʔωϧʹϩʔυ͢ΔϢʔβεϖʔεͷϓϩάϥϜ • ࣮ͭͰͳ͍ CQGUPPM
JQSPVUF CDD FUD • ͦΕͧΕඍົʹΈ͕ҧ͏ • ओͳࣄ &-'ϑΝΠϧ͔ΒϩʔυʹඞཁͳใΛऔΓग़͢ όΠτίʔυΛม͢Δ CQG Λͬͯϩʔυ͢Δ -
F#1'όΠφϦͷॻ͖͑ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 24 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ -
F#1'όΠφϦͷॻ͖͑ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 25 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit -
F#1'όΠφϦͷॻ͖͑ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 26 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit -
͜Ε$ͷϓϩάϥϜͰ͍͏ͱͲͷ෦ʁ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 27 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit count = bpf_map_lookup_elem(&pkt_counter, &index); -
͍͜͠ͷͰ݁ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 28 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit .BQͷϑΝΠϧσΟεΫϦϓλ .BQͷ࣮ΞυϨε -
.BQͷϑΝΠϧσΟεΫϦϓλΛखʹೖΕΔʹʁ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 29 • CQG #1'@."1@$3&"5&ίϚϯυΛ͏ • ͜ͷγεςϜίʔϧͷҾΛ&-'ϑΝΠϧ͔Β͔͖ूΊΔ
• CQGUPPMTͷ߹ lNBQTzηΫγϣϯ͔ΒҾΛͱͬͯ͘Δ struct { __u32 map_type; // Mapの”タイプ” (Array, Hash, Trie…) __u32 key_size; // Keyの⻑さ __u32 value_size; // Valueの⻑さ __u32 max_entries; // 最⼤要素数 __u32 map_flags; // フラグ __u32 inner_map_fd; // (Map in Mapという物がある…) __u32 numa_node; // どのNUMAノードでメモリを取るか char map_name[BPF_OBJ_NAME_LEN]; // 名前 }; -
$ͷίʔυ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 30 __attribute__((section(”maps”))) struct bpf_elf_map pkt_counter = {
.type = BPF_MAP_TYPE_ARRAY, .key_size = sizeof(uint32_t), .value_size = sizeof(uint32_t), .max_entries = 1 }; union bpf_attr { struct { /* BPF_MAP_CREATE */ __u32 map_type; __u32 key_size; __u32 value_size; __u32 max_entries; __u32 map_flags; __u32 inner_map_fd; __u32 numa_node; char map_name[BPF_OBJ_NAME_LEN]; }; lQLU@DPVOUFSz γϯϘϧ໊ -
Ͳ͏ͬͯGEΛόΠτίʔυʹຒΊࠐΉʁʁ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 31 r1 = 0 *(u32 *)(r10 -
4) = r1 r2 = r10 r2 += -4 r1 = 0 ll <= ここに埋め込む call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit • .BQͷϑΝΠϧσΟεΫϦϓλΛखʹೖΕΔํ๏Θ͔ͬͨ • Ͳ͏ͬͯຒΊࠐΉʁ • ຒΊࠐ͏ʹͲ͜ʹຒΊࠐΜͩΒ͍͍͔Θ͔Βͳ͍ͷͰʁ -
ϦϩέʔγϣϯηΫγϣϯΛݟͯΈΔ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 32 $ readelf -r demo.o Relocation section
'.rel.text' at offset 0x150 contains 1 entry: Offset Info Type Sym. Value Sym. Name 000000000020 000300000001 unrecognized: 1 0000000000000000 pkt_counter -
ϦϩέʔγϣϯηΫγϣϯΛݟͯΈΔ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 33 $ readelf -r demo.o Relocation section
'.rel.text' at offset 0x150 contains 1 entry: Offset Info Type Sym. Value Sym. Name 000000000020 000300000001 unrecognized: 1 0000000000000000 pkt_counter -
0GGTFUYʹԿ͕͋Δ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 34 $ llvm-objdump -disassemble demo.o test.o: file
format ELF64-BPF Disassembly of section .text: xdp_program: 0: b7 01 00 00 00 00 00 00 r1 = 0 1: 63 1a fc ff 00 00 00 00 *(u32 *)(r10 - 4) = r1 2: bf a2 00 00 00 00 00 00 r2 = r10 3: 07 02 00 00 fc ff ff ff r2 += -4 4: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll 6: 85 00 00 00 01 00 00 00 call 1 7: b7 01 00 00 01 00 00 00 r1 = 1 8: 15 00 03 00 00 00 00 00 if r0 == 0 goto +3 <LBB0_2> 9: b7 01 00 00 01 00 00 00 r1 = 1 10: c3 10 00 00 00 00 00 00 lock *(u32 *)(r0 + 0) += r1 11: b7 01 00 00 02 00 00 00 r1 = 2 LBB0_2: 12: bf 10 00 00 00 00 00 00 r0 = r1 13: 95 00 00 00 00 00 00 00 exit -
0GGTFUYʹԿ͕͋Δ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 35 $ llvm-objdump -disassemble demo.o test.o: file
format ELF64-BPF Disassembly of section .text: xdp_program: 0: b7 01 00 00 00 00 00 00 r1 = 0 1: 63 1a fc ff 00 00 00 00 *(u32 *)(r10 - 4) = r1 2: bf a2 00 00 00 00 00 00 r2 = r10 3: 07 02 00 00 fc ff ff ff r2 += -4 4: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll 6: 85 00 00 00 01 00 00 00 call 1 7: b7 01 00 00 01 00 00 00 r1 = 1 8: 15 00 03 00 00 00 00 00 if r0 == 0 goto +3 <LBB0_2> 9: b7 01 00 00 01 00 00 00 r1 = 1 10: c3 10 00 00 00 00 00 00 lock *(u32 *)(r0 + 0) += r1 11: b7 01 00 00 02 00 00 00 r1 = 2 LBB0_2: 12: bf 10 00 00 00 00 00 00 r0 = r1 13: 95 00 00 00 00 00 00 00 exit -
0GGTFUYʹԿ͕͋Δ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 36 • MEEX໋ྩ • CJUͷଈΛϨδελʹϩʔυ͢Δ • CJU͏
• ໋ྩʹԼҐCJU • ໋ྩʹ্ҐCJU • ͜͜ʹGEΛຒΊࠐΊΕ0, struct bpf_insn { __u8 opcode; __u8 dst_reg:4 __u8 src_reg:4 __s16 off; __s32 imm }; uint64_t foo = 0x12345; struct bpf_insn lddw[2] = { { BPF_LDDW, BPF_R1, 0, 0, (uint32_t)foo }, { 0 , 0, 0, 0, ((uint64_t)foo) >> 32 } }; -
ΧʔωϧͲ͏ͬͯGEΛݟ͚ͭΔʁʁ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 37 • MEEXී௨ʹCJUͷΛϩʔυ͢Δͷʹ͑Δ • ΧʔωϧͲ͏ͬͯGEͱී௨ͷΛݟ͚Δʁʁ • MEEXͷTSD@SFHͷͱ͜ΖʹಛผͳΛೖΕ͓ͯ͘
struct bpf_insn { __u8 opcode; __u8 dst_reg:4 __u8 src_reg:4 __s16 off; __s32 imm }; int map_fd = bpf(BPF_MAP_CREATE, &attr, sizeof(attr)); struct bpf_insn lddw[2] = { { BPF_LDDW, BPF_R1, BPF_PSEUDO_MAP_FD, 0, map_fd }, { 0 , 0, 0, 0, 0 } }; -
F#1' -PBEFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 38 • F#1'ϓϩάϥϜΛΧʔωϧʹϩʔυ͢ΔϢʔβεϖʔεͷϓϩάϥϜ • ࣮ͭͰͳ͍ CQGUPPM
JQSPVUF CDD FUD • ͦΕͧΕඍົʹΈ͕ҧ͏ • ओͳࣄ &-'ϑΝΠϧ͔ΒϩʔυʹඞཁͳใΛऔΓग़͢ όΠτίʔυΛม͢Δ CQG Λͬͯϩʔυ͢Δ -
F#1'ͷ$ϓϩάϥϜ͕࣮ߦ͞ΕΔ·Ͱ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 39 NIC driver bpf(2) bpftool demo.c clang
Verifier ここで動く! CQGUPPM ϩʔμ ͷ͘͠Έ 7FSJGJFSͷ͘͠Έ -
F#1' 7FSJGJFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 40 • F#1'ϓϩάϥϜ͕ΧʔωϧͰ҆શʹ࣮ߦͰ͖Δ͔੩తʹݕূ͢ΔΈ • ϧʔϓͷݕग़ ϝϞϦΞΫηεҧͷݕग़
0VUPGCPVOEKVNQͷݕग़ • ϙΠϯλϦʔΫͷݕग़ • FUD FUD FUD • ओͳࣄ • ݕূ • ϓϩάϥϜͷॻ͖͑ɾ࠷దԽ -
F#1' 7FSJGJFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 41 • F#1'ϓϩάϥϜ͕ΧʔωϧͰ҆શʹ࣮ߦͰ͖Δ͔੩తʹݕূ͢ΔΈ • ϧʔϓͷݕग़ ϝϞϦΞΫηεҧͷݕग़
0VUPGCPVOEKVNQͷݕग़ • ϙΠϯλϦʔΫͷݕग़ • FUD FUD FUD • ओͳࣄ • ݕূ • ϓϩάϥϜͷॻ͖͑ɾ࠷దԽ -
__attribute__((section("maps"))) struct bpf_elf_map pkt_counter = { .type = BPF_MAP_TYPE_ARRAY, .size_key
= sizeof(uint32_t), .size_value = sizeof(uint32_t), .max_elem = 1 }; __attribute__((section("xdp"))) int xdp_program(struct xdp_md *ctx) { uint32_t *count, index = 0; count = bpf_map_lookup_elem(&pkt_counter, &index); if (count == NULL) { return XDP_DROP; } __sync_fetch_and_add(count, 1); return XDP_PASS; } F#1'ϓϩάϥϜͷݕূ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 42 • ͜ͷνΣοΫΒͳ͔ͬͨΒͲ͏ͳΔͷ ͔ʁʁ • Լͷ@@TZOD@GFUDI@BOE@BEE/6--ࢀর ʹͳͬͯ͠·͏ͷ͔ʁʁ • ΧʔωϧࢮΜͰ͠·͏ͷ͔ʁ • 7FSJGJFSʹ͔ΕΔͷͰϩʔυͰ͖ͳ͍ • 7FSJGJFSͲ͏ͬͯͦΕΛνΣοΫ͍ͯ͠ Δͷ͔ʁ -
ϩʔυ͠Α͏ͱͯ͠ΈΔ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 43 # sudo bpftool prog load demo_invalid.o
/sys/fs/bpf/demo type xdp libbpf: load bpf program failed: Permission denied libbpf: -- BEGIN DUMP LOG --- libbpf: 0: (b7) r1 = 0 1: (63) *(u32 *)(r10 -4) = r1 2: (bf) r2 = r10 3: (07) r2 += -4 4: (18) r1 = 0xffff8b1353362800 6: (85) call bpf_map_lookup_elem#1 7: (b7) r1 = 1 8: (c3) lock *(u32 *)(r0 +0) += r1 R0 invalid mem access 'map_value_or_null' libbpf: -- END LOG -- libbpf: failed to load program 'xdp' libbpf: failed to load object 'demo_invalid.o' Error: failed to load object file -
3JOWBMJENFNBDDFTTbNBQ@WBMVF@PS@OVMM`ͷҙຯ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 44 • NBQ@WBMVF@PS@OVMMͱͳʹ͔ • 7FSJGJFS͕Ϩδελͷʹ͚͍ͭͯΔʮܕʯͷΑ͏ͳͷ • ʮNBQͷWBMVF͘͠OVMMʯOVMMͳՄೳੑ͕͋Δ
ࢀর͢Δͷةݥʂ int xdp_program(struct xdp_md *ctx) { uint32_t *count, index=0; count = bpf_map_lookup_elem(&pkt_counter, &index); if (count == NULL) { return XDP_DROP; } __sync_fetch_and_add(count, 1); return XDP_PASS; } 3lNBQ@WBMVF@PS@OVMMz 3lNBQ@WBMVFz ϩʔυͰ͖ͳ͍ϓϩάϥϜ ϩʔυͰ͖ΔϓϩάϥϜ int xdp_program(struct xdp_md *ctx) { uint32_t *count, index = 0; count = bpf_map_lookup_elem(&pkt_counter, &index); __sync_fetch_and_add(count, 1); return XDP_PASS; } -
3JOWBMJENFNBDDFTTbNBQ@WBMVF@PS@OVMM`ͷҙຯ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 45 • NBQ@WBMVF@PS@OVMMͱͳʹ͔ • 7FSJGJFS͕Ϩδελͷʹ͚͍ͭͯΔʮܕʯͷΑ͏ͳͷ • ʮNBQͷWBMVF͘͠OVMMʯOVMMͳՄೳੑ͕͋Δ
ࢀর͢Δͷةݥʂ 3lNBQ@WBMVF@PS@OVMMz 3lNBQ@WBMVFz ϩʔυͰ͖ͳ͍ϓϩάϥϜ ϩʔυͰ͖ΔϓϩάϥϜ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r0 = 2 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit -
F#1' 7FSJGJFS ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 46 • F#1'ϓϩάϥϜ͕ΧʔωϧͰ҆શʹ࣮ߦͰ͖Δ͔੩తʹݕূ͢ΔΈ • ϧʔϓͷݕग़ ϝϞϦΞΫηεҧͷݕग़
0VUPGCPVOEKVNQͷݕग़ • ϙΠϯλϦʔΫͷݕग़ • FUD FUD FUD • ओͳࣄ • ݕূ • ϓϩάϥϜͷॻ͖͑ɾ࠷దԽ -
7FSJGJFSʹΑΔόΠτίʔυͷॻ͖͑ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 47 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit -
7FSJGJFSʹΑΔόΠτίʔυͷ࠷దԽ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 48 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit DBMMͱʁ DBMMͷΠϯϥΠϯԽ -
DBMMͷҙຯ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 49 • )FMQFSGVODUJPOʹҰҙͷ*%͕͍͍ͭͯͯ CQG@NBQ@MPPLVQ@FMFN*% • DBMM໋ྩͷଈʹ*%ΛೖΕ͓ͯ͘ͱΧʔωϧͷதͰຊͷؔݺͼग़͠ʹม͞ΕΔ •
$ͰͷIFMQFSGVODUJPOFYUFSOؔͰͳؔ͘ϙΠϯλͷఆͱͯ͠ఆٛ͞Ε͍ͯΔ • ίϯύΠϧ͢Δͱ͏·͍۩߹ʹDBMMʹͳΔ enum bpf_func_id { BPF_FUNC_unspec, BPF_FUNC_map_lookup_elem, BPF_FUNC_map_update_elem, BPF_FUNC_map_delete_elem, BPF_FUNC_probe_read, ... }; static void *(*bpf_map_lookup_elem)(void *map, void *key) = (void *) BPF_FUNC_map_lookup_elem; -
CQG@NBQ@MPPLVQ@FMFNͷΠϯϥΠϯԽ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 50 • )FMQFS'VODUJPOͷॲཧͷʹΑͬͯF#1'ͷ໋ྩ͚ͩͰදݱͰ͖Δ • "SSBZ.BQͷMPPLVQͦͷ࠷ͨΔྫ • ΠϯϥΠϯԽͰੑೳΛՔ͙
static void * array_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_array *array = container_of(map, struct bpf_array, map); u32 index = *(u32)key; if (unlikely(index >= array->map.max_entries)) return NULL; return array->value + array->elem_size * (index & array->index_mask); } r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 -
CQG@NBQ@MPPLVQ@FMFNͷΠϯϥΠϯԽ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 51 • )FMQFS'VODUJPOͷॲཧͷʹΑͬͯF#1'ͷ໋ྩ͚ͩͰදݱͰ͖Δ • "SSBZ.BQͷMPPLVQͦͷ࠷ͨΔྫ • ΠϯϥΠϯԽͰੑೳΛՔ͙
static void * array_map_lookup_elem(struct bpf_map *map, void *key) { struct bpf_array *array = container_of(map, struct bpf_array, map); u32 index = *(u32)key; if (unlikely(index >= array->map.max_entries)) return NULL; return array->value + array->elem_size * (index & array->index_mask); } r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 -
7FSJGJFSʹΑΔόΠτίʔυͷ࠷దԽ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 52 r1 = 0 *(u32 *)(r10 -4)
= r1 r2 = r10 r2 += -4 r1 = map[id:22] r1 += 208 r0 = *(u32 *)(r2 +0) if r0 >= 0x1 goto pc+3 r0 <<= 3 r0 += r1 goto pc+1 r0 = 0 r1 = 1 if r0 == 0x0 goto pc+3 r1 = 1 lock *(u32 *)(r0 +0) += r1 r1 = 2 r0 = r1 exit r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 0 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit EFNPP ίϯύΠϧޙ CQGUPPM PVUQVU CQG ͷલ WFSJGJFSPVUQVU +*5લ r1 = 0 *(u32 *)(r10 - 4) = r1 r2 = r10 r2 += -4 r1 = 3 ll call 1 r1 = 1 if r0 == 0 goto +3 r1 = 1 lock *(u32 *)(r0 + 0) += r1 r1 = 2 r0 = r1 exit -
F#1'ͷ$ϓϩάϥϜ͕࣮ߦ͞ΕΔ·Ͱ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 53 NIC driver bpf(2) bpftool demo.c clang
Verifier ここで動く! __attribute__((section("maps"))) struct bpf_elf_map pkt_counter = { .type = BPF_MAP_TYPE_ARRAY, .size_key = sizeof(uint32_t), .size_value = sizeof(uint32_t), .max_elem = 1 }; __attribute__((section("xdp"))) int xdp_program(struct xdp_md *ctx) { uint32_t *count, index = 0; count = bpf_map_lookup_elem(&pkt_counter, &index); if (count == NULL) { return XDP_DROP; } __sync_fetch_and_add(count, 1); return XDP_PASS; } -
·ͱΊ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 54 • -JOVYͷF#1'ʹ͍ͭͯجຊతͳͱ͜ΖΛ͓͞Β͍ͨ͠ • ݴޠ༷ ''* .BQ
7FSJGJFSʜ • ϓϩάϥϜ͕ϩʔυ͞ΕΔ·ͰͷྲྀΕΛͬ͟ͱݟͨ • F#1'ϩʔμʔͷ͘͠Έ • 7FSJGJFSͷ͘͠Έ • ͜ͷ͕໌͔Βօ͞Μͷʹཱͭ͜ͱΛفΓ·͢ -
͜Μͳͷ·ͩ·ͩණࢁͷҰ֯ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 55 • ษڧʹͳΔࢿྉΛڍ͓͛ͯ͘ • ӳޠ • %JWFJOUP#1'BMJTUPGSFBEJOHNBUFSJBM
ૉΒ͍͠F#1'ؔ࿈ใͷϦϯΫू • $JMMJVNͷϚχϡΞϧ F#1'ΛͬͨωοτϫʔΫϓϩάϥϛϯάͳΒ͜͜Λࢀর • #1'1FSGPSNBODF5PPMT ʮৄղγεςϜύϑΥʔϚϯεʯͷ#SFOEBO(SFHHࢯͷ৽࡞ • ຊޠ • ਭෆ ࠓͨ͠Α͏ͳϚχΞοΫͳײ͡ͷF#1'ͷ͕ࡌ͍ͬͯΔ • !*5ͷ࿈ࡌ !*5ͷF#1'ʹؔ͢Δ࿈ࡌ ϢʔεέʔεΑΓJOUFSOBMͷ͕ଟ͍ -
એ ͖ͬͱ໌͔ΒཱͭF#1'ͷ͘͠Έ cૣါଠ࿕ 56 • ϚϧνϓϥοτϑΥʔϜͰ͑ΔF#1'ॲཧܥ αϒγεςϜͷ։ൃΛ͍ͯ͠·͢ • HFOFSJDFCQG IUUQTHJUIVCDPN:VUBSP)BZBLBXBHFOFSJDFCQG
• 'SFF#4%,FSOFM6TFS -JOVY,FSOFM6TFS .BD046TFS • ߦఔͷάϧʔίʔυͰF#1'͕ಈ͘ • *OUFSQSFUFS +*5GPSY • .BQ 7FSJGJFS FUD • ίϯτϦϏϡʔλɾϢʔεέʔεҊืूத