At Clear Link Technologies, we built a proprietary data platform that solves a specific problem: coordinating between providers using their own data, and helping customers make informed decisions based on that coordination. Since launch, we've scaled this platform to support multiple partner sites generating $150K in monthly recurring profit.
This isn't a theoretical post about SaaS. It's the actual story of what we built, the problems we solved, and what we'd do differently.
The Problem We Solved
Customers needed a way to coordinate across multiple providers using proprietary data. Before our platform, they had to manually:
- Collect data from multiple providers
- Reconcile inconsistencies and gaps
- Make decisions without a unified view
- Handle escalations and special cases manually
Our platform automated this coordination and gave them visibility into provider relationships using their own proprietary data.
Architecture Decisions
Backend: Golang + Gin
We chose Golang for its:
- Concurrency: Managing multiple provider connections simultaneously required handling thousands of concurrent requests
- Performance: Fast startup times and low memory footprint (important for elastic scaling)
- Simplicity: Easy to onboard engineers; Gin is lightweight compared to Rails or Django frameworks
Frontend: Vue.js + React
We started with Vue.js for rapid iteration on the UI. As the platform grew, we integrated React for more complex interactive features (dashboards, real-time updates). Both coexist in the same codebase.
Database: PostgreSQL for Structured Data
PostgreSQL handled our relational data (customers, providers, relationships). We used:
- JSONB columns for flexible provider-specific fields
- Custom indexes on frequently-queried columns
- Replication for read scaling
Graph Database: Neo4j (Later AWS Neptune)
Provider coordination is fundamentally a graph problem—providers connect to customers, customers to services, services to outcomes. Neo4j was perfect initially, but costs scaled with our data growth.
Real Scaling Challenges
Challenge 1: Managing Multiple Provider Connections
Each provider has a different API, different data formats, different latency characteristics. We built an abstraction layer that:
- Normalizes provider data into a common schema
- Retries failed requests with exponential backoff
- Caches provider responses to avoid rate limits
- Monitors provider uptime and alerts on degradation
Challenge 2: Real-Time Data Consistency
Customers need up-to-date provider data, but we can't query providers on every request (rate limits). We solved this with:
- Scheduled syncs: Batch refresh provider data every 15 minutes
- Event-driven updates: When a customer changes something, we immediately reflect it
- Staleness indicators: UI shows when data was last updated (transparency)
The Graph Database Story: Neo4j → AWS Neptune
Why Neo4j Became Expensive
As our customer touchpoint graph grew (millions of relationships), Neo4j licensing costs scaled with instance size:
- 2022 (early): $3K/month (small instance, 50GB graph)
- 2022 (late): $8K/month (medium instance, 150GB graph)
- 2023 (early): Projected $15K/month (licensing was accelerating)
The Migration (2022-2023)
We took a phased approach:
- Assessment: Identified all Cypher queries, understood access patterns
- Testing: Launched Neptune instance, ported 30+ queries to Gremlin, validated results
- Data migration: Exported graph from Neo4j, transformed, bulk-loaded into Neptune
- Parallel running: Ran both systems for 2 weeks, comparing results
- Cutover: Switched production traffic Friday night, monitored closely
Result: $25K annual savings, better uptime (managed service), no data loss.
Deployment Evolution: From Elastic Beanstalk to Zero-Downtime CI/CD
The Before: Direct Elastic Beanstalk Deployments
We deployed directly to EC2 Elastic Beanstalk multiple times per week. Each deployment:
- Caused brief downtime (seconds to minutes) while the service restarted
- Was manual and error-prone
- Had no automated rollback if something went wrong
- Required engineering team to be on-call during deploys
The After: Bitbucket + AWS CodePipeline
We built a fully automated CI/CD pipeline:
Git push to Bitbucket
↓
CodePipeline triggered
↓
CodeBuild: Build Docker image
↓
CodeDeploy: Deploy to green environment (ECS)
↓
Health checks pass
↓
Load balancer switches to green (instant, zero-downtime)
↓
Monitor for 5 minutes
↓
If healthy: Done. If issues: Rollback to blue
Now we deploy multiple times per week with zero downtime and full confidence.
Data Pipeline Evolution: From AWS SWF to Airflow
The Problem with AWS SWF
We started with AWS Simple Workflow Service (SWF) to orchestrate data pipelines:
- No visibility: Hard to debug when jobs failed
- Timeout issues: Long-running provider syncs would timeout
- Retry logic: Had to build custom retry handling (error-prone)
- Data validation: No built-in checks; errors propagated downstream
- Monitoring: Minimal alerting; we'd find out hours later about failures
Switching to Apache Airflow + Docker
We standardized on Airflow + Docker for 3-4 initial pipelines, then built more:
- Web UI: Instant visibility into pipeline status, logs, metrics
- DAG-based: Dependencies defined explicitly (no timeout surprises)
- Built-in retry: Exponential backoff, max retries, alerting
- Data validation: Integrated checks at each step
- Monitoring: Automatic alerts on failure
What We'd Do Differently
- Start with Airflow from day one: SWF taught us that orchestration is critical. Airflow's simplicity would have saved months
- Invest in provider API abstractions earlier: We added this after the first 3 providers. Starting with a clean abstraction would have saved refactoring
- Graph database architecture sooner: We realized the coordinate problem was a graph after building relational first. Starting with Neptune would have been cleaner
- CI/CD pipeline before scaling: Manual deployments became a bottleneck. Automating earlier would have freed up time for feature work
Key Lessons
1. Pick Tools That Eliminate Operational Pain
SWF and manual deployments required constant babysitting. Airflow and automated CI/CD let us sleep at night. The time saved pays for itself immediately.
2. Managed Services Scale Better Than DIY
Neo4j → Neptune: moving to a managed service eliminated ops burden and reduced costs. AWS handles backups, replication, monitoring.
3. Visibility Prevents Most Problems
Airflow's Web UI caught errors before they cascaded. SWF hid failures until they affected customers.
4. Standardization Enables Team Growth
Once we standardized on Airflow + Docker, onboarding new engineers was fast. They didn't need to understand SWF quirks; they just learned Airflow DAGs.
Conclusion
Building a provider coordination platform taught us that infrastructure investments compound over time. Every hour spent on CI/CD automation, data pipeline monitoring, and database optimization freed hours for actual feature development.
The $150K/month revenue didn't come from clever code—it came from a reliable, maintainable platform that customers could trust. That reliability came from investing in the unglamorous work of automation, monitoring, and operational excellence.