[{"data":1,"prerenderedAt":403},["ShallowReactive",2],{"project-mysql-load-test":3},{"id":4,"title":5,"body":6,"description":384,"duration":385,"extension":386,"featured":307,"github":387,"image":388,"live":389,"meta":390,"navigation":307,"order":172,"path":391,"role":389,"seo":392,"status":393,"stem":394,"team_size":389,"tech":395,"type":400,"year":401,"__hash__":402},"projects\u002Fproject\u002Fmysql-load-test.md","MySQL Load Test",{"type":7,"value":8,"toc":376},"minimark",[9,14,23,27,33,70,74,79,84,101,106,118,122,127,130,194,198,201,372],[10,11,13],"h2",{"id":12},"the-problem","The Problem",[15,16,17,18,22],"p",{},"Synthetic benchmarks like Sysbench are great for raw hardware testing but fail to represent ",[19,20,21],"strong",{},"real-world traffic patterns",". Production databases often face skewed data access (Pareto principle), specific query complexity distributions, and \"thundering herd\" scenarios that synthetic random queries cannot simulate.",[10,24,26],{"id":25},"my-solution","My Solution",[15,28,29,30,32],{},"I engineered ",[19,31,5],{},", a \"Record & Replay\" ecosystem designed to replicate production behavior with high fidelity.",[34,35,36,48,58,64],"ul",{},[37,38,39,42,43,47],"li",{},[19,40,41],{},"Traffic Capture:"," Instead of inventing queries, it parses real SQL traffic directly from network packets (",[44,45,46],"code",{},".pcap",") or TShark logs.",[37,49,50,53,54,57],{},[19,51,52],{},"Weighted Replay:"," It doesn't just run queries randomly; it calculates the \"fingerprint\" weight of every query type. If ",[44,55,56],{},"SELECT * FROM products"," happens 80% of the time in production, the load tester ensures it happens 80% of the time during the test.",[37,59,60,63],{},[19,61,62],{},"High Concurrency:"," Built on Go's goroutines to spawn thousands of concurrent virtual users with minimal memory footprint.",[37,65,66,69],{},[19,67,68],{},"Observability:"," Integrated Prometheus exporter to visualize Latency (p99, p95) and QPS in real-time.",[10,71,73],{"id":72},"technical-deep-dive","Technical Deep Dive",[75,76,78],"h3",{"id":77},"architecture-decisions","Architecture Decisions",[15,80,81],{},[19,82,83],{},"Why Packet Capture (Gopacket) over General Logs?",[34,85,86],{},[37,87,88,91,92,95,96,100],{},[19,89,90],{},"Zero Overhead:"," Enabling the \"General Query Log\" in MySQL kills performance. By capturing traffic at the network layer (TCP\u002FIP) using ",[44,93,94],{},"gopacket",", I could extract SQL queries from a production server without adding ",[97,98,99],"em",{},"any"," load to the database itself.",[15,102,103],{},[19,104,105],{},"Why Go?",[34,107,108],{},[37,109,110,113,114,117],{},[19,111,112],{},"Channel-Based Pipeline:"," The ",[44,115,116],{},"query-collector"," component uses a streaming pipeline pattern. It reads gigabytes of PCAP data, processes distinct query fingerprints, and writes them to the DB in parallel without loading the entire dataset into RAM.",[75,119,121],{"id":120},"key-features-i-built","Key Features I Built",[123,124,126],"h4",{"id":125},"_1-weighted-random-selection-strategy","1. Weighted Random Selection Strategy",[15,128,129],{},"To mimic real traffic, I implemented a logic that selects queries based on their historical frequency. The config allows fetching these weights dynamically from the database.",[131,132,137],"pre",{"className":133,"code":134,"language":135,"meta":136,"style":136},"language-yaml shiki shiki-themes github-light github-dark","# configuration snippet\nfingerprint_weights_query: |\n  SELECT\n    qf2.Hash AS Hash,\n    CAST(COUNT(*) AS DECIMAL(10,4)) \u002F qft.c * 100 AS Weight\n  FROM QueryFingerprint qf2\n  ...\n","yaml","",[44,138,139,148,163,170,176,182,188],{"__ignoreMap":136},[140,141,144],"span",{"class":142,"line":143},"line",1,[140,145,147],{"class":146},"sJ8bj","# configuration snippet\n",[140,149,151,155,159],{"class":142,"line":150},2,[140,152,154],{"class":153},"s9eBZ","fingerprint_weights_query",[140,156,158],{"class":157},"sVt8B",": ",[140,160,162],{"class":161},"szBVR","|\n",[140,164,166],{"class":142,"line":165},3,[140,167,169],{"class":168},"sZZnC","  SELECT\n",[140,171,173],{"class":142,"line":172},4,[140,174,175],{"class":168},"    qf2.Hash AS Hash,\n",[140,177,179],{"class":142,"line":178},5,[140,180,181],{"class":168},"    CAST(COUNT(*) AS DECIMAL(10,4)) \u002F qft.c * 100 AS Weight\n",[140,183,185],{"class":142,"line":184},6,[140,186,187],{"class":168},"  FROM QueryFingerprint qf2\n",[140,189,191],{"class":142,"line":190},7,[140,192,193],{"class":168},"  ...\n",[123,195,197],{"id":196},"_2-network-layer-extraction-pipeline","2. Network Layer Extraction Pipeline",[15,199,200],{},"The tool treats network packets as a stream of raw bytes, reassembles the TCP stream, and extracts the MySQL protocol payload. This allows \"sniffing\" queries even from uninstrumented legacy applications.",[131,202,206],{"className":203,"code":204,"language":205,"meta":136,"style":136},"language-bash shiki shiki-themes github-light github-dark","# From internal\u002Fcmd\u002Fquery-collector\u002Finput_pcap.go\nfunc (i *InputPcap) StartExtractor(ctx context.Context, out chan\u003C- *query.Query) error {\n    handle, _ := pcap.OpenOffline(i.file)\n    packetSource := gopacket.NewPacketSource(handle, handle.LinkType())\n\n    for packet := range packetSource.Packets() {\n        \u002F\u002F ... TCP reassembly and MySQL protocol decoding logic\n        out \u003C- extractedQuery\n    }\n}\n","bash",[44,207,208,213,259,282,303,309,317,346,360,366],{"__ignoreMap":136},[140,209,210],{"class":142,"line":143},[140,211,212],{"class":146},"# From internal\u002Fcmd\u002Fquery-collector\u002Finput_pcap.go\n",[140,214,215,219,222,226,229,232,235,238,241,244,247,250,253,256],{"class":142,"line":150},[140,216,218],{"class":217},"sScJk","func",[140,220,221],{"class":157}," (i ",[140,223,225],{"class":224},"sj4cs","*",[140,227,228],{"class":168},"InputPcap",[140,230,231],{"class":157},") StartExtractor(",[140,233,234],{"class":217},"ctx",[140,236,237],{"class":168}," context.Context,",[140,239,240],{"class":168}," out",[140,242,243],{"class":168}," chan",[140,245,246],{"class":161},"\u003C",[140,248,249],{"class":168},"-",[140,251,252],{"class":224}," *",[140,254,255],{"class":168},"query.Query",[140,257,258],{"class":157},") error {\n",[140,260,261,264,267,270,273,276,279],{"class":142,"line":165},[140,262,263],{"class":217},"    handle,",[140,265,266],{"class":168}," _",[140,268,269],{"class":168}," :=",[140,271,272],{"class":168}," pcap.OpenOffline",[140,274,275],{"class":157},"(",[140,277,278],{"class":217},"i.file",[140,280,281],{"class":157},")\n",[140,283,284,287,289,292,294,297,300],{"class":142,"line":172},[140,285,286],{"class":217},"    packetSource",[140,288,269],{"class":168},[140,290,291],{"class":168}," gopacket.NewPacketSource",[140,293,275],{"class":157},[140,295,296],{"class":217},"handle,",[140,298,299],{"class":168}," handle.LinkType",[140,301,302],{"class":157},"())\n",[140,304,305],{"class":142,"line":178},[140,306,308],{"emptyLinePlaceholder":307},true,"\n",[140,310,311,314],{"class":142,"line":184},[140,312,313],{"class":161},"    for",[140,315,316],{"class":157}," packet := range packetSource.Packets() {\n",[140,318,319,322,325,328,331,334,337,340,343],{"class":142,"line":190},[140,320,321],{"class":217},"        \u002F\u002F",[140,323,324],{"class":168}," ...",[140,326,327],{"class":168}," TCP",[140,329,330],{"class":168}," reassembly",[140,332,333],{"class":168}," and",[140,335,336],{"class":168}," MySQL",[140,338,339],{"class":168}," protocol",[140,341,342],{"class":168}," decoding",[140,344,345],{"class":168}," logic\n",[140,347,349,352,355,357],{"class":142,"line":348},8,[140,350,351],{"class":217},"        out",[140,353,354],{"class":161}," \u003C",[140,356,249],{"class":168},[140,358,359],{"class":168}," extractedQuery\n",[140,361,363],{"class":142,"line":362},9,[140,364,365],{"class":157},"    }\n",[140,367,369],{"class":142,"line":368},10,[140,370,371],{"class":157},"}\n",[373,374,375],"style",{},"html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .s9eBZ, html code.shiki .s9eBZ{--shiki-default:#22863A;--shiki-dark:#85E89D}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .szBVR, html code.shiki .szBVR{--shiki-default:#D73A49;--shiki-dark:#F97583}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}",{"title":136,"searchDepth":150,"depth":150,"links":377},[378,379,380],{"id":12,"depth":150,"text":13},{"id":25,"depth":150,"text":26},{"id":72,"depth":150,"text":73,"children":381},[382,383],{"id":77,"depth":165,"text":78},{"id":120,"depth":165,"text":121},"A high-performance load testing suite that captures real production traffic via PCAP analysis and replays it with weighted concurrency to accurately simulate database load.","1 months","md","https:\u002F\u002Fgithub.com\u002Fszuryuu\u002Fmysql-load-test","\u002Fimages\u002Fprojects\u002Fmysql.png",null,{},"\u002Fproject\u002Fmysql-load-test",{"title":5,"description":384},"Completed","project\u002Fmysql-load-test",[396,397,398,399],"Go","Gopacket","Prometheus","MySQL","Intern Project","2025","swZaS0GgHomHJY6mveSeGAbnlQ6N6qUpI2vilbsKtgI",1776582962960]