Back to Projects
2025completed

KRX Market Data Distribution System

Designed and built a production-grade market data distribution system that captures real-time financial data from exchange UDP multicast feeds, parses FIX protocol messages, and distributes structured data through a fault-tolerant message broker architecture. The system features 4-layer microservice isolation, asynchronous replica broadcasting with circuit breaker protection, and a time-range message replay engine for downstream client recovery.

Role

Lead Backend Engineer

Team

Solo Developer

Duration

6 months

Key Metrics

30K+ msg/s

Throughput

< 1ms

p99 Latency

99.9%

Uptime

13

Data Types

5

Microservices

30+

RabbitMQ Consumers

System Architecture

Raw FIXRAW FIXOptionalWrite LogsResend RAWOptionalParsed JSONREST + WSMetricsUDP MulticastFeedsExchange Market DataMDDS GatewayPacket CaptureSequence Detection200MB SO_RCVBUFLog FilesRaw Message LogsReplay ServiceRead from Log Files →Resend to RAW ChannelIndependent from MDDS GatewayRabbitMQRAW ChannelUnprocessed FIX dataParsed ChannelStructured JSON dataOptional 3rd PartyQueueMarket Data GatewayFIX ParserJSON Transform13 Message RoutersHandle Logic MDDSCacheRedisCaching / StorageAPI ServerConsumes Parsed dataAnalyticsReal-time processing3rd Party ClientsExternal subscribersYSradar TechnicalSystemDashboardTauri + Svelte 5GrafanaDATA INGESTIONMESSAGE BROKERCONSUMERSMONITORINGREPLAY SYSTEMLEGENDData flowOptional pathIndependent service
UDP Multicast → MDDS Gateway → RabbitMQ (RAW Channel) → Market Data Gateway (FIX parsing) → RabbitMQ (Parsed Channel) → Downstream Consumers. MDDS writes log files for Replay Service which resends to RAW channel independently.

Data Pipeline

1

UDP Multicast

KRX exchange raw FIX feeds

2

MDDS Gateway

Packet capture + sequencing

3

RAW Channel

Unprocessed FIX in RabbitMQ

4

Market Data GW

Parse + 13 message routers

5

Parsed Channel

Structured JSON in RabbitMQ

6

Consumers

API, Analytics, 3rd Parties

Performance At Scale

30K+

msg/s Throughput

Sustained processing without drops during peak trading

< 1ms

p99 Latency

UDP receive to RabbitMQ publish on parsed channel

Zero

UDP Drops

200MB SO_RCVBUF absorbs ATO/ATC burst spikes

< 1ms

GC Pause

Zero-allocation path keeps Go GC negligible

13

Message Routers

Dedicated FIX parser for each KRX data type

30+

Active Consumers

Processing across multiple downstream services

Screenshots

KRX MDDS System Dashboard
Real-time monitoring dashboard showing throughput, latency, queue depth, error rate, and message type distribution

Tech Stack

Core Runtime

  • Go 1.22
  • Zero-allocation design

Message Broker

  • RabbitMQ
  • 30+ consumers
  • Dual-channel

Cache & Storage

  • Redis (caching)
  • SQLite (metadata)

Monitoring

  • Prometheus
  • Grafana
  • Custom alert engine

Desktop App

  • Tauri 2 + Svelte 5
  • 9 screens
  • Cross-platform

Protocol

  • FIX Protocol
  • UDP Multicast
  • WebSocket

Key Features

014-layer fault-isolated process architecture with independent deployment
02Zero-allocation UDP multicast receiver with 200MB kernel buffer for burst absorption
03Dual-channel RabbitMQ architecture: RAW (unprocessed FIX) and Parsed (structured JSON)
04Custom FIX protocol parser with 13 specialized message type handlers
05File-based message replay engine with independent Replay Service
06FNV-1a symbol-based partitioning for guaranteed per-symbol ordering
07Rule-based alert engine with severity tiers and cooldown deduplication
08Cross-platform desktop dashboard (Tauri + Svelte 5) with real-time WebSocket
09Circuit breaker on async replica broadcasting to prevent cascade failures
10Prometheus metrics with custom histograms for throughput, latency, queue depth
11Redis-backed caching layer with TTL-based invalidation

Challenges & Solutions

Problem

UDP packet loss during ATO/ATC burst — thousands of messages in milliseconds

Solution

200MB kernel SO_RCVBUF + zero-allocation byte buffer reuse + FNV-1a partitioning. Zero drops under production stress testing.

Problem

Strict per-symbol message ordering across distributed consumers

Solution

FNV-1a hash partitioning ensures same stock symbol always routes to same consumer in exact order received.

Problem

Live replay for clients who missed data — without impacting production

Solution

Independent Replay Service reads log files, publishes to RAW channel. Fully decoupled from live Gateway — no shared state.

Problem

Real-time visibility across 5 microservices for operations team

Solution

Prometheus + Grafana + custom alert engine with severity tiers. Desktop dashboard (Tauri + Svelte 5) for native monitoring experience.

Technical Deep Dive

Zero-Allocation Data Path

UDP listener allocates a single buffer, reuses it across reads. FIX parser uses raw byte scanning (bytes.IndexByte) without string allocation — GC pause under 1ms at 30K msg/s.

Dual-Channel Broker Architecture

MDDS Gateway pushes unprocessed FIX to RAW channel. Market Data Gateway parses through 13 routers, publishes structured JSON to Parsed channel. Clients subscribe to either channel.

File-Based Replay Engine

Every raw message is logged to files. Replay Service reads logs and publishes to RAW channel — completely independent from live MDDS Gateway. No shared state, no performance impact.

Desktop Admin Dashboard

Tauri 2 + Svelte 5 native app with 9 screens: real-time throughput monitoring, message distribution charts, alert management, log viewer, and CLI replay reference.

Circuit Breaker & Fault Isolation

Async replica broadcasting with circuit breaker protection. Each microservice is fully isolated — if one crashes, others continue unaffected. Auto-recovery on reconnection.

Interested in similar work?

Let's discuss how I can help with your project.

View More ProjectsGet in Touch