Architecture overview
Social Explorer is a monorepo of cooperating services around one PostgreSQL database.
┌────────────┐ ┌──────────────────────────┐
Browser ──▶│ Web app │─────▶│ PostgreSQL + pg_duckdb │◀── ETL pipelines (Python)
│ (SvelteKit)│ │ (analytics tables/views) │◀── Radio service (WavePulse)
└─────┬──────┘ └──────────────────────────┘
│ Dapr
┌─────────┴──────────┐
│ Redis (state/pubsub)│── Crawler (Go: Telegram, YouTube)
└────────────────────┘Components
| Directory | Role |
|---|---|
ex-accell-ui/ | SvelteKit 2 / Svelte 5 web app and its API routes (TypeScript, Tailwind CSS 4) |
PostgreSQL + pg_duckdb | Single analytics database. Queries run through DuckDB's engine on PostgreSQL tables |
etl/ | Python pipelines that load datasets such as GDELT into the database |
crawler/ | Go service that crawls Telegram, YouTube and Bluesky, scheduled through Dapr jobs (source lives in a separate repository for now) |
wavepulse/ | Radio recording and transcription service, derived from NYU's WavePulse |
jupyter/ | Jupyter Lab environment for research notebooks |
deploy/, docker-compose.yml | Dapr component definitions and the full-stack compose file |
Web app
- Server code lives under
src/lib/server/(authentication, query error handling) andsrc/routes/api/(HTTP endpoints).src/hooks.server.tsauthenticates and authorises every request before routes run. - Analytics API.
/api/unified-analytics?type=...serves dashboards from the analytics database. A missing table returns empty data flaggedmeta.unavailableso unused modules stay quiet; real failures return500. - Workflows. The workflow builder generates Python Dapr workflows that admins can deploy and run. Publishing stores the definition and a generated script in
published_workflows; each run is a row inworkflow_executions. There is a single generator (pythonDaprWorkflowGenerator) and the hybrid executor runs what it produces. - Data sharing.
src/lib/server/dataSharing/builds Frictionless Data Packages from catalogued datasets and, whenIPFS_ENABLED=true,src/lib/services/ipfsService.tsruns a Helia IPFS node inside the app process to publish and fetch them;ipfs_published_packagesis the registry. Off by default and admin-only, because published data is public and permanent. See Data sharing over IPFS.
Data model in brief
Posts from every platform land in shared tables that analytics modules (sentiment, topics, language, engagement) build on, plus views that join them for cross-module analysis.
More detail
See the data model and crawler pages. A page on the workflow engine is planned.