diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 00000000..571a35c7 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,53 @@ +name: CI + +on: + push: + branches: [master-v1] + pull_request: + branches: [master-v1] + +concurrency: + group: ci-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + name: Typecheck, test & build + runs-on: ubuntu-latest + timeout-minutes: 30 + + steps: + - uses: actions/checkout@v4 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.2.12 + + - uses: actions/setup-node@v4 + with: + node-version: 20 + + # `bun install` runs the `prepare` hook, which generates the Prisma clients. + - name: Install dependencies + run: bun install --frozen-lockfile + + - name: Typecheck + run: bun run typecheck + + - name: Build + run: bun run build + env: + NODE_ENV: production + + # Non-blocking for now: `web-ui` carries pre-existing + # @typescript-eslint/no-explicit-any errors. Remove continue-on-error + # once those are cleared so lint becomes a required check. + - name: Lint + run: bun run lint + continue-on-error: true + + # Non-blocking until the mock-harness failures in the agent test suite + # are fixed. Remove continue-on-error once the suite is green. + - name: Test + run: bun run test + continue-on-error: true diff --git a/CHANGELOG.md b/CHANGELOG.md index 2297651b..57730afb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # Changelog -All notable changes to Nucleus Cloud Ops will be documented in this file. +All notable changes to Nucleus Ops will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). @@ -56,7 +56,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [1.0.0] - 2026-02-28 ### Added -- Initial release of Nucleus Cloud Ops +- Initial release of Nucleus Ops - AWS multi-account management - Automated resource scheduling (EC2, RDS, ECS) - AI DevOps agent with LangGraph diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 33d29393..e964f48d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,7 +25,7 @@ All types of contributions are encouraged and valued. See the [Table of Contents ## Code of Conduct This project and everyone participating in it is governed by the -[Nucleus Cloud Ops Code of Conduct](CODE_OF_CONDUCT.md). +[Nucleus Ops Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to the project maintainers. @@ -67,7 +67,7 @@ We use GitHub issues to track bugs and errors. If you run into an issue with the ### Suggesting Enhancements -This section guides you through submitting an enhancement suggestion for Nucleus Cloud Ops, **including completely new features and minor improvements to existing functionality**. Following these guidelines will help maintainers and the community to understand your suggestion and find related suggestions. +This section guides you through submitting an enhancement suggestion for Nucleus Ops, **including completely new features and minor improvements to existing functionality**. Following these guidelines will help maintainers and the community to understand your suggestion and find related suggestions. #### Before Submitting an Enhancement @@ -84,7 +84,7 @@ Enhancement suggestions are tracked as [GitHub issues](/issues). - Provide a **step-by-step description of the suggested enhancement** in as many details as possible. - **Describe the current behavior** and **explain which behavior you expected to see instead** and why. At this point you can also tell which alternatives you do not like. - You may want to include **screenshots and animated GIFs** which help you demonstrate the steps or point out the part which the suggestion is related to. -- **Explain why this enhancement would be useful** to most Nucleus Cloud Ops users. You may also want to point out the other projects that solved it better and which could serve as inspiration. +- **Explain why this enhancement would be useful** to most Nucleus Ops users. You may also want to point out the other projects that solved it better and which could serve as inspiration. ### Your First Code Contribution @@ -95,6 +95,71 @@ Enhancement suggestions are tracked as [GitHub issues](/issues). 5. Push to the branch (`git push origin feature/amazing-feature`) 6. Open a Pull Request +#### Setting up your development environment + +Nucleus Ops is an **Nx 21 + Bun** monorepo. Requires Node.js 20+, Bun, and Docker. + +```bash +bun install # single install at the root, hoists to all apps +cp .env.example .env # one root .env, loaded by both apps +docker compose up -d postgres # local PostgreSQL with pgvector +cd apps/web-ui && bun run db:migrate # apply migrations +``` + +Run the two services (in separate terminals): + +```bash +bun run dev # web-ui on http://localhost:3001 +bun run dev:workers # pg-boss background workers +``` + +#### Before opening a pull request + +Always invoke tasks through the root scripts so Nx caching and project graph work: + +```bash +bun run lint # ESLint across all projects +bun run typecheck # TypeScript, no emit +bun run test # Vitest (web-ui + workers) and Jest (root) +bun run build # verify both apps build +bun run e2e # Playwright E2E (starts the dev server automatically) +``` + +`bun run verify` runs typecheck, test, and build for **affected** projects only — a +faster pre-push check on a focused change. + +> **Note:** the `web-ui` project currently has pre-existing `@typescript-eslint/no-explicit-any` +> errors. Please don't add new ones, but don't feel obliged to fix unrelated ones in your PR. + +#### Repository layout + +| Path | Contains | +|---|---| +| `apps/web-ui/` | Next.js App Router — pages, API routes, services, LangGraph agents | +| `apps/workers/` | pg-boss background jobs (scheduler, discovery, right-sizing, kb-sync) | +| `apps/web-ui-e2e/` | Playwright end-to-end tests | +| `libs/prisma/` | Prisma schema and migrations, shared by both apps | +| `infra/` | Pulumi stacks (`networking` → `compute`) and CI/CD specs | +| `docs/` | Architecture docs, PRDs, and screenshots | + +#### Conventions that will come up in review + +- **Data access:** never call Prisma directly from an API route or service — go through the + repository factory (`@/lib/db/repository-factory`). +- **Multi-tenancy:** always scope queries with `getTenantClient(tenantId)`. `$executeRaw` is + *not* intercepted by the tenant extension — scope it manually. +- **Authorization:** every mutating API route calls `authorize(action, Subject)` from + `@/lib/rbac/authorize`. +- **Auditing:** anything that modifies an AWS resource must be logged via `AuditService`. +- **Frontend:** server state uses TanStack Query hooks in `lib/queries/`, forms use React Hook + Form + Zod, toasts import `toast` from `sonner`. Don't hand-roll `useEffect` + `fetch`. +- **UI primitives** in `components/ui/` are shadcn/ui-generated — consume them, don't modify them. +- **Naming:** components are `kebab-case.tsx` exporting `PascalCase` named exports; services are + `kebab-case-service.ts`. + +Database schema changes go in `libs/prisma/schema.prisma`, followed by +`bun run db:migrate` in `apps/web-ui` and `bun run db:generate` in **both** apps. + ## Styleguides ### Commit Messages diff --git a/LICENSE b/LICENSE index 348feca4..1e9d5068 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,6 @@ MIT License -Copyright (c) 2024 Nucleus Cloud Ops +Copyright (c) 2024-2026 Kartik Manimuthu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.MD b/README.MD deleted file mode 100644 index 39a36f4a..00000000 --- a/README.MD +++ /dev/null @@ -1,309 +0,0 @@ -# Nucleus Cloud Ops - -

- Nucleus Cloud Ops Dashboard -

- - -

- AWS Cloud Operations Platform with AI Ops & GenAI DevOps Agent built on AWS Bedrock for Multi-Account Management. -

- -

- Features • - Quick Start • - Documentation • - Architecture • - Contributing -

- ---- - -## 🎯 What is Nucleus Cloud Ops? - -Nucleus Cloud Ops is a centralized platform for automating AWS resource management across multiple accounts. It leverages **AI Ops** and **GenAI** to optimize cloud operations and reduce costs. - -### The Problem - -- **Idle Resources**: Development and staging environments run 24/7 but are only used during business hours -- **Manual Management**: DevOps teams spend hours manually managing resource states -- **Multi-Account Complexity**: Enterprises struggle to coordinate schedules across dozens of AWS accounts -- **Lack of Visibility**: No centralized view of scheduling operations and cost savings - - -### The Solution - -Nucleus Ops provides: - -- 🕐 **Automated Scheduling**: Start/stop EC2, RDS, and ECS resources on custom schedules -- 🌐 **Multi-Account Support**: Manage N accounts from a single pane of glass -- 🤖 **AI-Powered Agent**: Natural language interface for cloud operations -- 📊 **Comprehensive Auditing**: Full visibility into all actions and executions -- 💰 **Cost Tracking**: Real-time savings estimation and reporting - ---- - -## ✨ Features - -### 📊 Cost Optimization Dashboard -Real-time overview of schedules, accounts, savings, and recent activity. - -### 🔗 AWS Account Integration -Seamlessly connect multiple AWS accounts using secure cross-account IAM roles. - -![AWS Accounts](docs/screenshots/aws_accounts_page.png) - -### ⏰ Intelligent Scheduling -Create flexible schedules with: -- Customizable time windows -- Weekday/weekend configurations -- Timezone support -- Multi-resource selection (EC2, RDS, ECS) - -### 📝 Audit Trail -Complete audit logging with: -- User and system action tracking -- 30-day retention -- Advanced filtering and search -- Export capabilities - -![Audit Logs](docs/screenshots/audit_logs_page.png) - -### 🤖 AI DevOps Agent -Natural language cloud management powered by Claude 4.5 Sonnet: -- Cost analysis and reporting -- Security audits -- Troubleshooting assistance -- Automated documentation -- **Multimodal support**: Upload images for visual analysis, diagram interpretation, and screenshot troubleshooting - -![AI Agent](docs/screenshots/ai_agent_page_final.png) - -### 🎨 Customizable UI -Dark/light themes, multiple color schemes, and typography options. - -![Settings](docs/screenshots/settings_page_final.png) - ---- - -## 🚀 Quick Start - -### Prerequisites - -- Node.js 22+ -- AWS CLI configured -- AWS CDK installed (`npm install -g aws-cdk`) -- Access to an AWS account with admin permissions - -### Installation - -```bash -# Clone the repository -git clone https://github.com/kartikmanimuthu/nucleus-cloud-ops.git -cd nucleus-cloud-ops - -# Install dependencies -npm install -cd web-ui && npm install && cd .. - -# Configure environment -cp web-ui/.env.local.example web-ui/.env.local -# Edit .env.local with your settings - -# Deploy infrastructure -npx cdk deploy --all - -# Or run locally -cd web-ui -npm run dev -``` - -### Environment Variables - -```bash -# Required -AWS_REGION=ap-south-1 -DYNAMODB_TABLE_NAME=nucleus-ops-main -DYNAMODB_AUDIT_TABLE_NAME=nucleus-ops-audit -NEXTAUTH_SECRET=your-secret-here - -# Optional (for AI Agent) -DYNAMODB_CHECKPOINT_TABLE=nucleus-ops-checkpoints -DYNAMODB_WRITES_TABLE=nucleus-ops-writes -TAVILY_API_KEY=your-tavily-key -``` - ---- - -## 📚 Documentation - -| Document | Description | -|----------|-------------| -| [Architecture Guide](docs/ARCHITECTURE.md) | Technical architecture, diagrams, and data models | -| [Features Guide](docs/FEATURES.md) | Comprehensive feature documentation with screenshots | -| [Multimodal Support](docs/MULTIMODAL_SUPPORT.md) | Image upload and multimodal AI capabilities | -| [Product Requirements](docs/product-requirement-document.md) | Original PRD | -| [Technical Review](docs/technical-review-document.md) | Technical decisions and rationale | -| [Schema Design](docs/schema-design.md) | DynamoDB table design | - ---- - -## 🏗️ Architecture - -``` -┌─────────────────────────────────────────────────────────────┐ -│ Nucleus Cloud Ops Platform │ -├─────────────────────────────────────────────────────────────┤ -│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ -│ │ Next.js │ │ LangGraph │ │ Lambda Scheduler │ │ -│ │ Web UI │ │ AI Agent │ │ (EventBridge) │ │ -│ └──────────────┘ └──────────────┘ └──────────────────┘ │ -│ │ │ │ │ -│ └────────────────┼────────────────────┘ │ -│ │ │ -│ ┌────────▼────────┐ │ -│ │ DynamoDB │ │ -│ │ (Single Table) │ │ -│ └─────────────────┘ │ -└─────────────────────────────────────────────────────────────┘ - │ - ┌────────▼────────┐ - │ AWS STS │ - │ AssumeRole │ - └────────┬────────┘ - │ - ┌────────────────────┼────────────────────┐ - │ │ │ -┌───────▼───────┐ ┌────────▼────────┐ ┌───────▼───────┐ -│ AWS Account │ │ AWS Account │ │ AWS Account │ -│ A │ │ B │ │ C │ -│ (EC2, RDS) │ │ (ECS, RDS) │ │ (EC2) │ -└───────────────┘ └─────────────────┘ └───────────────┘ -``` - -### Technology Stack - -| Layer | Technologies | -|-------|--------------| -| **Frontend** | Next.js 15, React 19, Tailwind CSS, Radix UI | -| **Backend** | Node.js, AWS SDK v3, LangGraph, LangChain | -| **AI** | AWS Bedrock (Claude 4.5 Sonnet) | -| **Database** | Amazon DynamoDB (Single Table Design) | -| **Infrastructure** | AWS CDK, Lambda, ECS Fargate, CloudFront | -| **Auth** | NextAuth.js, AWS Cognito | - ---- - -## 📂 Project Structure - -``` -nucleus-cloud-ops/ -├── bin/ # CDK app entry point -├── lib/ # CDK stack definitions -│ ├── cdkStack.ts # Main stack -│ ├── computeStack.ts # ECS, ALB, CloudFront -│ ├── networkingStack.ts # VPC, Subnets -│ └── webUIStack.ts # Web UI deployment -├── lambda/ # Lambda functions -│ └── scheduler/ # Scheduler execution logic -├── web-ui/ # Next.js application -│ ├── app/ # App router pages -│ ├── components/ # React components -│ ├── lib/ # Utilities and services -│ │ └── agent/ # AI agent implementation -│ └── styles/ # CSS styles -├── docs/ # Documentation -│ ├── ARCHITECTURE.md # Technical architecture -│ ├── FEATURES.md # Feature guide -│ └── screenshots/ # UI screenshots -└── test/ # Test files -``` - ---- - -## 🔐 Security - -- **Cross-Account Access**: Uses temporary credentials via `sts:AssumeRole` -- **Least Privilege**: Minimal IAM permissions for target accounts -- **Encryption**: DynamoDB encryption at rest (KMS) -- **Authentication**: NextAuth.js with Cognito integration -- **Audit Logging**: All actions recorded with 30-day retention - ---- - -## 🛠️ Development - -### Running Locally - -```bash -cd web-ui - -# With AWS profile -AWS_PROFILE=your-profile npm run dev - -# Access at http://localhost:3000 -``` - -### Deploying - -```bash -# Deploy all stacks -npx cdk deploy --all --profile your-profile - -# Deploy specific stack -npx cdk deploy WebUIStack --profile your-profile -``` - -### Testing - -```bash -# Unit tests (Vitest — web-ui) -cd web-ui && npm run test - -# Unit tests (Jest — CDK stacks) -npm test - -# Lint -cd web-ui && npm run lint - -# E2E tests (Playwright — requires dev server) -npx playwright test # Run all E2E tests -npx playwright test tests/e2e/accounts.spec.ts # AWS Accounts module (60 tests) -npx playwright test tests/e2e/navigation.spec.ts # App navigation flows -npx playwright test tests/e2e/marketing.spec.ts # Marketing/landing page -npx playwright test tests/e2e/docs.spec.ts # Documentation pages -npx playwright test --headed # Run in headed mode (see the browser) -npx playwright test --ui # Interactive UI mode / debugger -npx playwright show-report # View last HTML report -``` - ---- - -## 🤝 Contributing - -Contributions are welcome! Please read our contributing guidelines before submitting PRs. - -1. Fork the repository -2. Create a feature branch (`git checkout -b feature/amazing-feature`) -3. Commit changes (`git commit -m 'Add amazing feature'`) -4. Push to branch (`git push origin feature/amazing-feature`) -5. Open a Pull Request - ---- - -## 📄 License - -This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. - ---- - -## 🙏 Acknowledgments - -- [AWS CDK](https://aws.amazon.com/cdk/) - Infrastructure as Code -- [LangGraph](https://github.com/langchain-ai/langgraph) - AI Agent Framework -- [Radix UI](https://www.radix-ui.com/) - Accessible UI Components -- [Tailwind CSS](https://tailwindcss.com/) - Utility-first CSS - ---- - -

Built with ❤️ for the cloud community

diff --git a/README.md b/README.md new file mode 100644 index 00000000..573d1de3 --- /dev/null +++ b/README.md @@ -0,0 +1,262 @@ +
+ +# Nucleus Ops + +**Open-source, self-hostable AI Ops agent for AWS.** + +Schedule idle resources across every account, discover your whole footprint, and run cloud operations in plain English — on infrastructure you own. + +[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) +[![Stars](https://img.shields.io/github/stars/kartikmanimuthu/nucleus-cloud-ops?style=flat)](https://github.com/kartikmanimuthu/nucleus-cloud-ops/stargazers) +[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](CONTRIBUTING.md) +[![Next.js](https://img.shields.io/badge/Next.js-15-black?logo=next.js)](https://nextjs.org/) +[![Pulumi](https://img.shields.io/badge/IaC-Pulumi-8A3391?logo=pulumi)](https://www.pulumi.com/) +[![PostgreSQL](https://img.shields.io/badge/DB-PostgreSQL-4169E1?logo=postgresql&logoColor=white)](https://www.postgresql.org/) + +[Quick Start](#-quick-start) · [Features](#-features) · [How it works](#%EF%B8%8F-how-it-works) · [Docs](#-documentation) · [Comparison](#-how-it-compares) · [Contributing](CONTRIBUTING.md) + +Nucleus Ops AI Ops agent running a cost analysis across AWS accounts + +
+ +--- + +## What is Nucleus Ops? + +Nucleus Ops is a multi-tenant AWS cloud operations platform you run yourself. It combines three things most teams cobble together separately: + +1. **A cost scheduler** that starts and stops non-production EC2, RDS, and ECS resources on a schedule across every connected account. +2. **An inventory + right-sizing engine** that continuously discovers what you actually have running. +3. **An AI Ops agent** (LangGraph + AWS Bedrock) that can investigate, explain, and act on your infrastructure through natural language — with approval gates and a full audit trail. + +Your AWS credentials never leave your environment. Cross-account access is exclusively via `sts:AssumeRole` — no long-lived keys are stored anywhere in the system. + +### Why it exists + +- Non-production environments run 24/7 but are used ~50 hours a week. Shutting them off outside business hours removes roughly **70% of their compute cost**. +- Once you pass a handful of AWS accounts, "what is running, and who changed it" stops being answerable from the console. +- Cloud-ops SaaS tools want cross-account access to your entire estate. For a lot of teams that is a non-starter — so this is self-hosted by design. + +--- + +## ✨ Features + +| | | +|---|---| +| ⏰ **Cost Scheduler** | Start/stop EC2, RDS, and ECS on cron schedules with timezone and weekday/weekend windows. | +| 🌐 **Multi-Account** | Manage many AWS accounts from one pane of glass via cross-account IAM roles. | +| 🤖 **AI Ops Agent** | Natural-language cloud operations on AWS Bedrock (Claude Sonnet 4.6). Plan → Execute → Reflect → Revise, with per-tool approval gates. | +| ⚡ **Agent Ops** | Scheduled and event-triggered autonomous agent runs (cron, Slack, Jira, API) with full run history. | +| 🔍 **Inventory Discovery** | Background scans across accounts and regions; filter, export, and ask questions about your footprint. | +| 📉 **Right-Sizing** | CloudWatch-driven EC2/RDS/EBS/ASG recommendations with live pricing data. | +| 📚 **Knowledge Base** | pgvector-backed RAG over S3, Bitbucket, Confluence, and inline docs. | +| 🔗 **Channels** | Slack slash commands, Jira automation triggers, Telegram, and MCP servers. | +| 🔐 **RBAC + Multi-Tenancy** | CASL-based per-module roles; every query is tenant-scoped at the ORM layer. | +| 📝 **Audit Trail** | Immutable log of every action, with filtering and CSV export. | +| 🔑 **Certificate Manager** | ACM certificate inventory, expiry monitoring, and cross-account deployment. | +| 🧠 **Agent Memory & Skills** | Semantic, episodic, and procedural memory; reusable, human-approved skills. | + +> Screenshots for each module live in [`docs/screenshots/`](docs/screenshots/) and the [Features guide](docs/FEATURES.md). + +--- + +## 🚀 Quick Start + +**Prerequisites:** Node.js 20+, [Bun](https://bun.sh), Docker (for local Postgres), and an AWS profile with access to Bedrock. + +```bash +# 1. Clone +git clone https://github.com/kartikmanimuthu/nucleus-cloud-ops.git +cd nucleus-cloud-ops + +# 2. Install (single Bun workspace — hoists to all apps) +bun install + +# 3. Configure — one root .env is shared by both apps +cp .env.example .env +# Required: DATABASE_URL, AWS_REGION, NEXTAUTH_SECRET, +# COGNITO_USER_POOL_ID, COGNITO_APP_CLIENT_ID, COGNITO_APP_CLIENT_SECRET + +# 4. Start Postgres (pgvector) and apply migrations +docker compose up -d postgres +cd apps/web-ui && bun run db:migrate && cd ../.. + +# 5. Run +bun run dev # web UI on http://localhost:3001 +bun run dev:workers # background job workers (separate terminal) +``` + +Then open , connect your first AWS account, and create a schedule. + +### Common commands + +```bash +bun run dev # web-ui (Next.js, port 3001) +bun run dev:workers # pg-boss workers +bun run build # build all projects via Nx +bun run test # run all test suites +bun run lint # lint all projects +bun run e2e # Playwright end-to-end suite +bun run graph # visualise the Nx task graph +``` + +--- + +## 🏗️ How it works + +```mermaid +flowchart TB + subgraph host["Your AWS account"] + CF["CloudFront + ALB"] + subgraph ecs["ECS Fargate"] + WEB["web-ui — Next.js
UI, API routes, LangGraph agents"] + WRK["workers — pg-boss
scheduler, discovery, right-sizing, KB sync"] + end + PG[("PostgreSQL + pgvector (RDS)")] + S3[("S3")] + BR["AWS Bedrock
Claude Sonnet 4.6"] + end + + subgraph TA["Connected AWS accounts"] + A1["Account A — EC2, RDS"] + A2["Account B — ECS, RDS"] + A3["Account C — EC2"] + end + + CF --> WEB + WEB <--> PG + WRK <--> PG + WEB --> BR + WEB --> S3 + WEB -- "sts:AssumeRole" --> TA + WRK -- "sts:AssumeRole" --> TA +``` + +Two services, one database. The Next.js app serves the UI, the REST API, and runs the LangGraph agents in-process; the workers service runs every scheduled and background job through pg-boss. All state lives in PostgreSQL (via Prisma) or S3 — there are **no Lambda functions and no DynamoDB**. + +### Stack + +| Layer | Technology | +|---|---| +| **Frontend** | Next.js 15, React 19, Tailwind CSS, Radix UI, TanStack Query, React Hook Form + Zod | +| **AI Agent** | LangGraph, LangChain, AWS Bedrock (Claude Sonnet 4.6), MCP | +| **Backend** | Node.js 20, AWS SDK v3, NextAuth.js + Cognito, CASL RBAC | +| **Database** | PostgreSQL with pgvector, Prisma ORM (repository pattern) | +| **Background jobs** | pg-boss workers on ECS Fargate | +| **Infrastructure** | Pulumi (`infra/networking` → `infra/compute`), ECS Fargate, RDS, CloudFront | +| **Monorepo** | Nx 21 + Bun workspaces | +| **Testing** | Vitest, Jest, Playwright, fast-check | + +--- + +## 📂 Repository layout + +``` +nucleus-cloud-ops/ +├── apps/ +│ ├── web-ui/ # Next.js App Router — pages, API routes, agents, services +│ ├── workers/ # pg-boss background jobs (scheduler, discovery, right-sizing, kb-sync) +│ └── web-ui-e2e/ # Playwright end-to-end tests +├── libs/ +│ └── prisma/ # Prisma schema + migrations (shared by both apps) +├── infra/ +│ ├── networking/ # Pulumi: VPC, subnets +│ ├── compute/ # Pulumi: ECS, RDS, Cognito, CloudFront +│ └── cicd/ # CodeBuild specs +└── docs/ # Architecture, features, PRDs, screenshots +``` + +--- + +## ☁️ Deploying to AWS + +Infrastructure is managed entirely by [Pulumi](https://www.pulumi.com/). Always deploy networking before compute. + +```bash +cd infra/networking && npm install && pulumi install +cd ../compute && npm install && pulumi install + +# Preview first +cd infra/compute && AWS_PROFILE=your-profile pulumi preview --stack prod + +# Deploy — networking, then compute +cd ../networking && AWS_PROFILE=your-profile pulumi up --stack prod --yes +cd ../compute && AWS_PROFILE=your-profile pulumi up --stack prod --yes +``` + +`pulumi up` detects source changes, builds and pushes the `web-ui` and `workers` Docker images, and rolls out new ECS task definitions automatically. Full guide: [`infra/DEPLOYMENT.md`](infra/DEPLOYMENT.md). + +--- + +## 🔐 Security + +- **No stored credentials.** All cross-account access uses temporary credentials from `sts:AssumeRole`. +- **Least privilege.** Target accounts grant a scoped IAM role; the CloudFormation template for it is generated in-app. +- **Tenant isolation.** Every query goes through `getTenantClient(tenantId)`, which enforces `tenant_id` scoping at the Prisma layer. +- **RBAC.** Module-scoped roles via CASL, checked on every mutating API route. +- **Audit logging.** Every action that touches an AWS resource is recorded immutably. +- **Approval gates.** The AI agent asks before running mutating operations unless you explicitly enable auto-approve. + +Found a vulnerability? Please follow [SECURITY.md](SECURITY.md) — do not open a public issue. + +--- + +## 🔍 How it compares + +| | Nucleus Ops | AWS Instance Scheduler | Cloud Custodian | Komiser | +|---|---|---|---|---| +| Self-hosted | ✅ | ✅ | ✅ | ✅ | +| Resource scheduling | ✅ | ✅ | ✅ (policy-driven) | ❌ | +| Multi-account inventory | ✅ | ❌ | ⚠️ via policies | ✅ | +| Right-sizing recommendations | ✅ | ❌ | ❌ | ⚠️ basic | +| AI agent / natural-language ops | ✅ | ❌ | ❌ | ❌ | +| Web UI with RBAC | ✅ | ❌ (config files) | ❌ (YAML/CLI) | ✅ | +| Slack / Jira integration | ✅ | ❌ | ⚠️ via hooks | ❌ | +| Audit trail | ✅ | ⚠️ CloudWatch | ⚠️ logs | ❌ | + +Cloud Custodian is an excellent policy engine and Komiser is strong at visibility — Nucleus Ops targets the operational layer above both: a UI, an agent, approvals, and an audit trail for a team, not just a CLI for an engineer. + +--- + +## 📚 Documentation + +| Document | Description | +|---|---| +| [Architecture](docs/ARCHITECTURE.md) | System design, data flow, and component boundaries | +| [AI Ops Architecture](docs/AI_OPS_ARCHITECTURE.md) | Agent graphs, tools, memory, and skills | +| [Features](docs/FEATURES.md) | Full feature walkthrough with screenshots | +| [Deployment](infra/DEPLOYMENT.md) | Production deployment via Pulumi | +| [Agent memory](docs/agent-memory-architecture.md) | Semantic, episodic, and procedural memory design | +| [Right-sizing](docs/RIGHT_SIZING_ARCHITECTURE.md) | Recommendation engine and pricing data | +| [Data model](libs/prisma/schema.prisma) | Prisma schema — the single source of truth for all models | +| [Tech stack](docs/techstack.md) | Dependency-level breakdown | + +The app also serves its own documentation site at `/docs` when running. + +--- + +## 🤝 Contributing + +Contributions are very welcome — see [CONTRIBUTING.md](CONTRIBUTING.md) for the development workflow, coding conventions, and commit style. Good places to start: + +- Browse [open issues](https://github.com/kartikmanimuthu/nucleus-cloud-ops/issues) +- Improve documentation (docs live in `docs/` and `apps/web-ui/content/docs/`) +- Add support for another AWS resource type in the scheduler or discovery scan + +Please also read our [Code of Conduct](CODE_OF_CONDUCT.md). + +--- + +## 📄 License + +MIT — see [LICENSE](LICENSE). Free to self-host, fork, and modify. + +--- + +
+ +Built for teams who want cloud automation without handing over the keys. + +⭐ Star the repo if this is useful to you. + +
diff --git a/SECURITY.md b/SECURITY.md index 974f56fc..931f602c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -2,22 +2,46 @@ ## Supported Versions -Use this section to tell people about which versions of your project are -currently being supported with security updates. +Nucleus Ops is developed on the `master-v1` branch. Security fixes are applied to the +latest release; older tags are not backported. | Version | Supported | | ------- | ------------------ | -| 1.0.x | :white_check_mark: | -| < 1.0 | :x: | +| Latest release / `master-v1` | :white_check_mark: | +| Older tags | :x: | ## Reporting a Vulnerability -We take security seriously. If you discover a security vulnerability within Nucleus Cloud Ops, please follow these steps: +We take security seriously. If you discover a security vulnerability in Nucleus Ops, +please follow these steps: -1. **Do NOT create a public GitHub issue.** Security vulnerabilities should be handled discreetly to protect users. -2. Send an email to **kartikmanimuthu@gmail.com**. -3. Include a detailed description of the vulnerability, steps to reproduce it, and any potential impact. -4. We will acknowledge your report within 48 hours and provide an estimated timeline for a fix. -5. Once the vulnerability is resolved, we will publish a security advisory and credit you for the discovery (if desired). +1. **Do NOT create a public GitHub issue.** Security vulnerabilities should be handled + privately until a fix is available. +2. Report it privately using GitHub's + [Report a vulnerability](https://github.com/kartikmanimuthu/nucleus-cloud-ops/security/advisories/new) + flow, or email **kartikmanimuthu@gmail.com**. +3. Include a detailed description, steps to reproduce, the affected version or commit, + and the potential impact. +4. You will receive an acknowledgement within 48 hours and an estimated timeline for a fix. +5. Once resolved, we will publish a security advisory and credit you for the discovery + (unless you prefer to stay anonymous). -Thank you for helping keep Nucleus Cloud Ops secure! +Please do not run automated scanners against infrastructure you do not own. Because +Nucleus Ops is self-hosted, any deployment you find on the internet belongs to a third +party — test only against your own instance. + +## Security Model + +A few properties worth knowing when assessing this project: + +- **No long-lived AWS credentials.** Cross-account access uses `sts:AssumeRole` to obtain + temporary credentials at request time. +- **Tenant isolation** is enforced in the data layer via `getTenantClient(tenantId)`, which + scopes every Prisma query by `tenant_id`. Note that raw `$executeRaw` calls bypass this + extension and must scope manually — a known sharp edge worth reviewing in contributions. +- **RBAC** is enforced per module with CASL on every mutating API route. +- **Audit logging** records every action that modifies an AWS resource. +- **Agent tool calls** are gated behind human approval unless auto-approve is explicitly + enabled by the operator. + +Thank you for helping keep Nucleus Ops secure. diff --git a/apps/web-ui/app/page.tsx b/apps/web-ui/app/page.tsx index 6aa2170a..52ca8f59 100644 --- a/apps/web-ui/app/page.tsx +++ b/apps/web-ui/app/page.tsx @@ -15,7 +15,7 @@ const features = [ { icon: Bot, title: "AI Ops Agent", - description: "Natural language cloud operations powered by Claude 4.6 Sonnet via AWS Bedrock. Plan → Execute → Reflect → Revise with auto-approve mode.", + description: "Natural language cloud operations powered by Claude Sonnet 4.6 via AWS Bedrock. Plan → Execute → Reflect → Revise with auto-approve mode.", }, { icon: Zap, @@ -92,7 +92,7 @@ export default function MarketingPage() { Sign in - + Get Started @@ -101,24 +101,26 @@ export default function MarketingPage() { {/* Hero */}
-
- - Open Source · MIT License -
-
- - Powered by AWS Bedrock + Claude 4.6 Sonnet +
+ + + Open Source · MIT License + + + + AWS Bedrock + Claude Sonnet 4.6 +

- AWS Cloud Ops,
- Automated & Intelligent + Self-hosted AI Ops
+ for AWS

- Stop paying for idle cloud resources. Nucleus Ops automates EC2, RDS, and ECS scheduling across all your AWS accounts — with an AI agent that speaks your language. Free and open source. + Stop paying for idle cloud resources. Nucleus Ops automates EC2, RDS, and ECS scheduling across all your AWS accounts — with an AI agent that speaks your language. Free, open source, and running on infrastructure you own.

- - Start for free + + Deploy your own {[ - { value: "70%", label: "avg. cost reduction" }, - { value: "91+", label: "AWS accounts" }, - { value: "31K+", label: "resources discovered" }, + { value: "~70%", label: "off idle non-prod compute" }, + { value: "91", label: "AWS accounts in production" }, + { value: "31K+", label: "resources under management" }, { value: "MIT", label: "open source license" }, ].map((stat) => (
@@ -145,6 +147,9 @@ export default function MarketingPage() {
))}
+

+ Account and resource counts are from the maintainer’s production deployment. Savings reflect shutting non-production resources outside a 50-hour work week. +

{/* Demo screenshot placeholder */} @@ -154,7 +159,7 @@ export default function MarketingPage() {
- nucleus-ops.io/app/dashboard + localhost:3001/app/dashboard
{[ @@ -214,108 +219,51 @@ export default function MarketingPage() { Proudly Open Source
-

Free forever. No hidden costs.

+

The whole product. MIT licensed.

- Nucleus Ops is MIT licensed. Self-host it on your own AWS account, contribute to the codebase, or fork it for your needs. + There is no paid tier, no feature gate, and no “open core” split. Everything you see here is in the repository — self-host it on your own AWS account, contribute to it, or fork it.

-
- {[ - { - name: "Self-Hosted", - price: "$0", - period: "forever", - description: "Deploy to your own AWS account", - features: [ - "All features included", - "Unlimited AWS accounts", - "Unlimited schedules", - "AI Ops Agent", - "Slack & Jira integration", - "Knowledge Base", - "Full source code access", - "MIT License", - ], - cta: "Deploy Now", - highlight: false, - }, - { - name: "Cloud Hosted", - price: "Coming", - period: "soon", - description: "Managed hosting — no infra to maintain", - features: [ - "All self-hosted features", - "Managed infrastructure", - "Automatic updates", - "99.9% uptime SLA", - "Backups included", - "Priority support", - ], - cta: "Join Waitlist", - highlight: true, - }, - { - name: "Enterprise", - price: "Custom", - period: "contact us", - description: "For large-scale operations", - features: [ - "Everything in Cloud Hosted", - "SSO / SAML", - "Custom SLA", - "Dedicated support", - "On-premise deployment", - "Custom integrations", - ], - cta: "Contact Us", - highlight: false, - }, - ].map((plan) => ( -
+
    + {[ + "Every feature included", + "Unlimited AWS accounts", + "Unlimited schedules and agent runs", + "AI Ops Agent + Agent Ops", + "Inventory discovery & right-sizing", + "Knowledge Base (pgvector RAG)", + "Slack, Jira, Telegram & MCP", + "RBAC, multi-tenancy & audit logs", + "Full source code access", + "No telemetry, no phone-home", + ].map((feat) => ( +
  • + + {feat} +
    • + ))} +
+
+ -
-
- {plan.name} -
-
- {plan.price} - - /{plan.period} - -
-

- {plan.description} -

-
-
    - {plan.features.map((feat) => ( -
  • - - {feat} -
    • - ))} -
- - {plan.cta} - -
- ))} + Read the install guide + + + + Browse the source + +
+

+ A managed hosting option may come later for teams who’d rather not run it themselves. It would run this same MIT-licensed codebase — self-hosting will never be the crippled tier. +

@@ -324,10 +272,10 @@ export default function MarketingPage() {

Ready to cut your AWS bill?

-

Connect your first AWS account in under 5 minutes. Free and open source.

+

Clone it, run it, and connect your first AWS account in under 5 minutes. Free and open source.

- - Get started free + + Deploy your own UI, API routes, LangGraph agents"] + WRK["workers — pg-boss
scheduler, discovery, right-sizing, KB sync"] + end + PG[("PostgreSQL + pgvector (RDS)")] + S3[("S3")] + BR["AWS Bedrock"] + end + + subgraph TA["Connected AWS accounts"] + A1["Account A"] + A2["Account B"] + A3["Account C"] + end + + CF --> WEB + WEB <--> PG + WRK <--> PG + WEB --> BR + WEB --> S3 + WEB -- "sts:AssumeRole" --> TA + WRK -- "sts:AssumeRole" --> TA +``` + +## The two services + +### `web-ui` + +A Next.js App Router application that serves three things from one container: + +- **Pages** under `app/app/` — the dashboard, inventory, schedules, agent workspace, and settings +- **REST API** under `app/api/` — every route authorises with CASL RBAC and reads through the repository layer +- **AI agents** — LangGraph state machines that run *in-process*, so a chat request and its tool calls share one runtime + +### `workers` + +A single Node process running [pg-boss](https://github.com/timgit/pg-boss) jobs. Because pg-boss uses PostgreSQL itself as the queue, there is no separate broker to operate. Jobs include: + +| Job | What it does | +|---|---| +| `scheduler` | Evaluates schedules and starts/stops resources across accounts | +| `discovery` | Multi-account, multi-region resource scans via the AWS SDK | +| `right-sizing` | CloudWatch-driven recommendations plus a weekly pricing refresh | +| `kb-sync` | Ingests and embeds knowledge-base sources (S3, Bitbucket, Confluence) | +| `agent-ops-scheduler` | Fires scheduled autonomous agent runs | +| `certificate-expiry-monitor` | Flags ACM certificates nearing expiry | + +## Where state lives + +Everything persistent is in PostgreSQL or S3. + +| Data | Storage | +|---|---| +| Accounts, schedules, RBAC, tenants | PostgreSQL via Prisma repositories | +| Inventory and right-sizing results | PostgreSQL | +| Audit log | PostgreSQL (immutable, with retention) | +| Agent chat history | PostgreSQL (`agent_chat_history`) | +| LangGraph thread checkpoints | PostgreSQL, with S3 offload for large checkpoints | +| Long-term agent memory + KB chunks | PostgreSQL with pgvector (HNSW index) | + +## The agent layer + +Three agent types share a common toolset and are constructed through `graph-factory.ts`: + +- **Fast agent** — a textbook ReAct loop for direct questions +- **Planning agent** — planner → executor → reflector → reviser for multi-step work +- **Deep agent** — extended thinking for long-running investigations + +On top of the graphs sit the cognitive features: semantic memory (recall/remember), episodic memory (past runs replayed as few-shot experience), procedural memory (operating rules learned from corrections), and Skills — reusable, human-approved playbooks. + +A chat turn is routed by a triage step that decides between answering directly and running a full graph, and can auto-select a relevant Skill. + +## Cross-account access + +Nucleus Ops never stores AWS credentials for the accounts it manages. When it needs to act on a connected account it calls `sts:AssumeRole` against a role that account has granted, and uses the resulting temporary credentials for that operation only. + +You create that role from a CloudFormation template generated in the app under **AWS Accounts → Integrate Account**. Scope it to the permissions you actually want Nucleus Ops to have — read-only if you only want inventory and recommendations. + +## Multi-tenancy + +Every tenant's data lives in the same tables, separated by `tenant_id`. Isolation is enforced in the data layer rather than by convention: application code obtains a client via `getTenantClient(tenantId)`, a Prisma extension that injects the tenant filter into every query. + + +`$executeRaw` is **not** intercepted by the tenant extension. Any raw SQL must scope +`tenant_id` manually. This is the one place where tenant isolation depends on the author +rather than the framework. + + +## Deployment topology + +Two Pulumi stacks, deployed in order: + +1. **`infra/networking`** — VPC, public and private subnets, subnet groups +2. **`infra/compute`** — ECS Fargate cluster running both services, RDS PostgreSQL, Cognito, S3, and CloudFront + +See [Installation](/docs/installation) for the deploy commands. + + +There are **no AWS Lambda functions** in this project. Everything that was once a Lambda — +scheduling, discovery, knowledge-base sync — is now a pg-boss job in the `workers` service. +Likewise, DynamoDB has been fully replaced by PostgreSQL. + diff --git a/apps/web-ui/content/docs/configuration.mdx b/apps/web-ui/content/docs/configuration.mdx index 633a3861..211344c6 100644 --- a/apps/web-ui/content/docs/configuration.mdx +++ b/apps/web-ui/content/docs/configuration.mdx @@ -81,10 +81,10 @@ Nucleus Ops supports standard cron syntax: ![AI Ops](/screenshots/ai-ops.png) -The AI Ops agent uses a **Plan → Execute → Reflect → Revise** loop powered by Claude 4.6 Sonnet via AWS Bedrock. +The AI Ops agent uses a **Plan → Execute → Reflect → Revise** loop powered by Claude Sonnet 4.6 via AWS Bedrock. **Controls:** -- **Model** — Claude 4.6 Sonnet (default) +- **Model** — Claude Sonnet 4.6 (default) - **Mode** — Fast (ReAct) or Planning - **Auto-approve tools** — Execute actions without confirmation - **Select Accounts** — Scope the agent to specific accounts diff --git a/apps/web-ui/content/docs/faq.mdx b/apps/web-ui/content/docs/faq.mdx index 11ce8697..959d0a31 100644 --- a/apps/web-ui/content/docs/faq.mdx +++ b/apps/web-ui/content/docs/faq.mdx @@ -11,7 +11,7 @@ Nucleus Ops is an open source AWS Cloud Operations Platform for multi-account re ### What AWS resources does Nucleus Ops support? -EC2 instances, RDS clusters (Aurora, MySQL, PostgreSQL), ECS services, and ElastiCache. The Inventory Discovery feature auto-discovers 30+ resource types including ACM certificates, Lambda functions, S3 buckets, and more. +Scheduling covers EC2 instances, RDS clusters and instances (Aurora, MySQL, PostgreSQL), ECS services, and Auto Scaling groups. Inventory Discovery auto-discovers a much wider set of resource types — ACM certificates, Lambda functions, S3 buckets, load balancers, and more. ### Does Nucleus Ops store my AWS credentials? @@ -51,7 +51,7 @@ The savings estimate is based on on-demand pricing for the resource type and reg ### What model does the AI agent use? -AWS Bedrock with Claude 4.6 Sonnet (`anthropic.claude-sonnet-4-6`). Streaming is enabled for real-time responses. The model selector in the UI lets you switch models. +AWS Bedrock with Claude Sonnet 4.6 (`anthropic.claude-sonnet-4-6`). Streaming is enabled for real-time responses. The model selector in the UI lets you switch models. ### What is the Plan → Execute → Reflect → Revise loop? @@ -67,7 +67,7 @@ Yes, with your approval. By default, the agent proposes actions and asks for con ### Is conversation history saved? -Yes. Conversations are stored in DynamoDB with configurable TTL (via `DYNAMODB_CHECKPOINT_TABLE`). Chat history is accessible from the history panel in the AI Ops interface. +Yes. Conversations are stored in PostgreSQL (the `agent_chat_history` table) with a configurable TTL, and LangGraph thread state is checkpointed to the same database. Chat history is accessible from the session sidebar in the AI Ops interface. ### What are Agent Skills? @@ -79,7 +79,7 @@ Pre-built prompt templates for common tasks like "Vital check of EC2 instances" ### How does inventory discovery work? -An ECS-based discovery Lambda scans all connected accounts and regions, normalizes resource data, and stores it in DynamoDB. Results are also pushed to S3 for vector indexing. +A background discovery job (pg-boss, running in the `workers` ECS service) scans all connected accounts and regions in parallel using the AWS SDK, normalizes the resource data, and stores it in PostgreSQL. Embeddings are written to pgvector for semantic search. ### How often is inventory refreshed? @@ -99,7 +99,7 @@ NextAuth.js with AWS Cognito. All sessions are JWT-based with configurable expir ### Is data encrypted at rest? -Yes. DynamoDB tables use AWS KMS encryption at rest. S3 buckets use server-side encryption. +Yes. The RDS PostgreSQL instance uses AWS KMS encryption at rest, and S3 buckets use server-side encryption. Secrets are held in AWS Secrets Manager, and Pulumi state is encrypted with KMS. ### Can I restrict which users can create schedules? @@ -111,18 +111,20 @@ Yes. Nucleus Ops has role-based access control (RBAC). Admins can assign `viewer ### Can I self-host Nucleus Ops? -Yes. Deploy with AWS CDK (ECS Fargate) or run locally with `npm run dev`. See [Installation](/docs/installation). +Yes — self-hosting is the primary way to run it. Deploy to your own AWS account with Pulumi (ECS Fargate), or run it locally with `bun run dev`. See [Installation](/docs/installation). -### What does the CDK deployment cost? +### What does the deployment cost? -Roughly $30–80/month for a small deployment (ECS Fargate + DynamoDB + CloudFront). Actual costs depend on usage and the number of accounts being scanned. +Roughly $70–150/month for a small production deployment (two ECS Fargate services, an RDS PostgreSQL instance, CloudFront, and NAT). Running locally against a Docker Postgres costs nothing beyond Bedrock inference. Actual costs depend on usage, retention, and how many accounts you scan. ### What's the tech stack? | Layer | Tech | |-------|------| -| Frontend | Next.js 15, React 19, Tailwind CSS, Radix UI | -| AI Agent | LangGraph, LangChain, AWS Bedrock (Claude 4.6 Sonnet), MCP | -| Infrastructure | AWS CDK v2, ECS Fargate, CloudFront, DynamoDB | +| Frontend | Next.js 15, React 19, Tailwind CSS, Radix UI, TanStack Query | +| AI Agent | LangGraph, LangChain, AWS Bedrock (Claude Sonnet 4.6), MCP | +| Database | PostgreSQL + pgvector, Prisma ORM | +| Background jobs | pg-boss workers (ECS Fargate) | +| Infrastructure | Pulumi, ECS Fargate, RDS, CloudFront | | Auth | NextAuth.js + AWS Cognito | -| Testing | Vitest (web-ui), Jest (CDK) | +| Testing | Vitest (web-ui, workers), Jest (root), Playwright (E2E) | diff --git a/apps/web-ui/content/docs/getting-started.mdx b/apps/web-ui/content/docs/getting-started.mdx index f1ea6c5a..fa6da3f3 100644 --- a/apps/web-ui/content/docs/getting-started.mdx +++ b/apps/web-ui/content/docs/getting-started.mdx @@ -6,8 +6,8 @@ description: Set up Nucleus Ops and connect your first AWS account in under 5 mi ## Prerequisites - An AWS account with admin permissions -- Node.js 22+ -- AWS CLI configured locally +- Node.js 20+ and [Bun](https://bun.sh) +- AWS CLI configured locally with a named profile ## Step 1: Sign In @@ -62,7 +62,7 @@ Example prompts: > "Vital check of my EC2 instances" -The agent uses a Plan → Execute → Reflect → Revise loop with Claude 4.6 Sonnet. Enable **Auto-approve** to let it execute actions without confirmation. +The agent uses a Plan → Execute → Reflect → Revise loop with Claude Sonnet 4.6. Enable **Auto-approve** to let it execute actions without confirmation. ## Next Steps diff --git a/apps/web-ui/content/docs/index.mdx b/apps/web-ui/content/docs/index.mdx index 39e0af5c..ee6b0a87 100644 --- a/apps/web-ui/content/docs/index.mdx +++ b/apps/web-ui/content/docs/index.mdx @@ -19,7 +19,7 @@ The dashboard gives you a real-time view of your entire AWS footprint — total |---------|-------------| | Cost Scheduler | Auto start/stop EC2, RDS, ECS on custom schedules — [configure](/docs/configuration#cost-scheduler) | | AWS Accounts | Multi-account management with cross-account IAM — [configure](/docs/configuration#aws-accounts) | -| AI Ops | Natural language agent powered by Claude 4.6 Sonnet — [configure](/docs/configuration#ai-ops-agent) | +| AI Ops | Natural language agent powered by Claude Sonnet 4.6 — [configure](/docs/configuration#ai-ops-agent) | | Agent Ops | Background agent runs via Slack, Jira, API, schedule — [tutorial](/docs/tutorials#tutorial-5-monitor-agent-execution-history) | | Inventory Discovery | Auto-discover all resources across all accounts — [tutorial](/docs/tutorials#tutorial-1-save-on-devstaging-environments) | | Knowledge Base | Vector-powered Q&A over your infrastructure docs — [tutorial](/docs/tutorials#tutorial-6-build-a-knowledge-base-from-your-docs) | @@ -31,8 +31,10 @@ The dashboard gives you a real-time view of your entire AWS footprint — total ## Quick Links - [Getting Started](/docs/getting-started) — Set up your first account in 5 minutes -- [Installation](/docs/installation) — Deploy Nucleus Ops to your AWS environment +- [Installation](/docs/installation) — Run locally or deploy to your AWS account with Pulumi +- [How It Works](/docs/architecture) — Services, data flow, and where the AI agent runs - [Configuration](/docs/configuration) — Configure accounts, schedules, and the AI agent +- [Security Model](/docs/security) — Credentials, tenant isolation, RBAC, and agent gating - [Slack Integration](/docs/slack-integration) — Slash commands, bot token, and channel setup - [Jira / Atlassian Integration](/docs/jira-integration) — Jira Automation trigger channel + MCP tool setup - [Tutorials](/docs/tutorials) — Step-by-step guides for common workflows diff --git a/apps/web-ui/content/docs/installation.mdx b/apps/web-ui/content/docs/installation.mdx index 5ba95df6..be240ddf 100644 --- a/apps/web-ui/content/docs/installation.mdx +++ b/apps/web-ui/content/docs/installation.mdx @@ -1,65 +1,132 @@ --- title: Installation -description: Deploy Nucleus Ops to your AWS environment using CDK or run it locally. +description: Run Nucleus Ops locally, or deploy it to your own AWS account with Pulumi. --- -## Option A: Run Locally +Nucleus Ops is a Bun + Nx monorepo containing two deployable services: + +- **`web-ui`** — the Next.js app (UI, REST API, and the LangGraph AI agents) +- **`workers`** — the pg-boss background job runner (scheduling, discovery, right-sizing, knowledge-base sync) + +Both share one PostgreSQL database and one root `.env` file. + +## Prerequisites + +| Requirement | Notes | +|---|---| +| Node.js 20+ | The container images run Node 20 | +| [Bun](https://bun.sh) | Workspace package manager and Nx task runner | +| Docker | For the local PostgreSQL (pgvector) instance | +| AWS CLI + a named profile | Needs access to Amazon Bedrock | +| [Pulumi CLI](https://www.pulumi.com/docs/install/) | Only for deploying to AWS | + +## Option A: Run locally ```bash git clone https://github.com/kartikmanimuthu/nucleus-cloud-ops.git cd nucleus-cloud-ops -npm install -cd web-ui && npm install +# Install once at the root — Bun workspaces hoist to every app +bun install + +# Single root .env, loaded by both apps +cp .env.example .env + +# Start PostgreSQL with the pgvector extension +docker compose up -d postgres -cp web-ui/.env.local.example web-ui/.env.local -# Edit .env.local with your settings +# Create the schema +cd apps/web-ui && bun run db:migrate && cd ../.. -cd web-ui -AWS_PROFILE=your-profile npm run dev +# Run the app and the workers in separate terminals +bun run dev # http://localhost:3001 +bun run dev:workers ``` -Open [http://localhost:3000](http://localhost:3000). +Open [http://localhost:3001](http://localhost:3001). -## Option B: Deploy to AWS with CDK + +`bun install` runs `prisma generate` automatically. After changing +`libs/prisma/schema.prisma`, re-run `bun run db:generate` inside both +`apps/web-ui` and `apps/workers`. + -### Prerequisites +### Environment variables + +The full annotated list lives in `.env.example`. The minimum needed to boot: + +| Variable | Required | Description | +|----------|----------|-------------| +| `DATABASE_URL` | ✅ | PostgreSQL connection string. Append `?connection_limit=10` in production. | +| `AWS_REGION` | ✅ | Primary AWS region (e.g. `ap-south-1`) | +| `NEXTAUTH_SECRET` | ✅ | Random secret used to sign session JWTs | +| `NEXTAUTH_URL` | ✅ | Base URL of the app (e.g. `http://localhost:3001`) | +| `COGNITO_USER_POOL_ID` | ✅ | Cognito user pool for authentication | +| `COGNITO_APP_CLIENT_ID` | ✅ | Cognito app client ID | +| `COGNITO_APP_CLIENT_SECRET` | ✅ | Cognito app client secret | +| `ASK_AI_GENERATION_MODEL` | — | Bedrock model for the agent (defaults to Claude Sonnet 4.6) | +| `BEDROCK_MODEL_ID` | — | Embedding model (must be 1024-dimensional) | +| `TAVILY_API_KEY` | — | Enables the agent's web-search tool | +| `LANGFUSE_*` | — | Optional LLM observability | + +## Option B: Deploy to AWS with Pulumi + +Infrastructure lives in two Pulumi stacks. **Networking must be deployed before compute** — the compute stack imports its VPC and subnet outputs. ```bash -npm install -g aws-cdk +# Install stack dependencies (first time only) +cd infra/networking && npm install && pulumi install +cd ../compute && npm install && pulumi install ``` -### Deploy +Preview before you apply: ```bash -# Bootstrap CDK (first time only) -npx cdk bootstrap --profile your-profile +cd infra/compute && AWS_PROFILE=your-profile pulumi preview --stack prod +``` + +Then deploy, in order: -# Deploy all stacks -npx cdk deploy --all --profile your-profile +```bash +cd infra/networking && AWS_PROFILE=your-profile pulumi up --stack prod --yes +cd ../compute && AWS_PROFILE=your-profile pulumi up --stack prod --yes ``` -This deploys: -- **NetworkingStack** — VPC, subnets, security groups -- **ComputeStack** — ECS Fargate, ALB, CloudFront -- **WebUIStack** — Next.js app container +This provisions: -### Environment Variables +- **networking** — VPC, public/private subnets, and subnet groups +- **compute** — ECS Fargate cluster running the `web-ui` and `workers` services, RDS PostgreSQL, Cognito, S3, and CloudFront -| Variable | Required | Description | -|----------|----------|-------------| -| `AWS_REGION` | ✅ | AWS region (e.g. `ap-south-1`) | -| `DYNAMODB_TABLE_NAME` | ✅ | Main DynamoDB table | -| `DYNAMODB_AUDIT_TABLE_NAME` | ✅ | Audit log table | -| `NEXTAUTH_SECRET` | ✅ | Random secret for NextAuth | -| `NEXTAUTH_URL` | ✅ | Your app URL | -| `DYNAMODB_CHECKPOINT_TABLE` | Optional | AI agent checkpoints | -| `TAVILY_API_KEY` | Optional | Web search for AI agent | +`pulumi up` hashes the source directories and uses the digest as the image tag, so changing `apps/web-ui/`, `apps/workers/`, or `libs/prisma/` automatically rebuilds the affected Docker image, pushes it to ECR, registers a new ECS task definition, and rolls out the service. Database migrations run on container start via the entrypoint. + + +There are no AWS Lambda functions in this project. Everything that used to be a +Lambda — scheduling, discovery, knowledge-base sync — now runs as a pg-boss job +inside the `workers` ECS service. + + +### Verify the deployment + +```bash +cd infra/compute && AWS_PROFILE=your-profile pulumi stack output --stack prod +``` + +1. The CloudFront URL from the stack output returns HTTP 200 +2. Both ECS services show `running == desired` task counts +3. No errors in the CloudWatch task logs for the five minutes after rollout + +### Rollback + +```bash +git revert HEAD +cd infra/compute && AWS_PROFILE=your-profile pulumi up --stack prod +``` ## Docker +The `web-ui` image installs dependencies with Bun and builds under `node:20-slim`. The runtime must be Node, not Bun — stdio-based MCP servers require `node`/`npx` on the PATH. + ```bash -cd web-ui -docker build -t nucleus-ops . -docker run -p 3000:3000 --env-file .env.local nucleus-ops +docker build -f apps/web-ui/Dockerfile -t nucleus-ops-web . +docker run -p 3000:3000 --env-file .env nucleus-ops-web ``` diff --git a/apps/web-ui/content/docs/meta.json b/apps/web-ui/content/docs/meta.json index 5491ca4b..742cc78c 100644 --- a/apps/web-ui/content/docs/meta.json +++ b/apps/web-ui/content/docs/meta.json @@ -4,7 +4,9 @@ "index", "getting-started", "installation", + "architecture", "configuration", + "security", "slack-integration", "jira-integration", "tutorials", diff --git a/apps/web-ui/content/docs/security.mdx b/apps/web-ui/content/docs/security.mdx new file mode 100644 index 00000000..bdee3fac --- /dev/null +++ b/apps/web-ui/content/docs/security.mdx @@ -0,0 +1,71 @@ +--- +title: Security Model +description: How Nucleus Ops handles AWS credentials, tenant isolation, authorization, and agent actions. +--- + +Nucleus Ops is designed to be given access to production AWS accounts, so it's worth being explicit about what it does and does not do with that access. If you are evaluating whether to run this, read this page first. + +## AWS credentials + +**Nucleus Ops never stores credentials for the AWS accounts it manages.** + +Access works through `sts:AssumeRole`. Each connected account grants a role that the Nucleus Ops task role is allowed to assume; at request time the platform assumes it, receives short-lived credentials, performs the operation, and discards them. There are no access keys for target accounts anywhere in the database, environment, or logs. + +Practical consequences: + +- Revoking access is a one-line IAM change in the target account — no coordination needed. +- The blast radius of a compromised Nucleus Ops instance is bounded by the trust policies and permissions of the roles you granted. +- **Scope the role to what you actually need.** If you only want inventory and right-sizing, grant read-only permissions. The scheduler needs start/stop permissions; nothing needs delete. + +The CloudFormation template for the role is generated in-app under **AWS Accounts → Integrate Account** — read it before you deploy it. + +## Authentication + +Authentication is handled by NextAuth.js, backed by AWS Cognito (with an optional credentials provider for local development). Sessions are JWT-based with configurable expiry. Because you self-host, your user directory never leaves your environment. + +## Authorization (RBAC) + +Every mutating API route calls `authorize(action, Subject)` before doing anything. Authorization uses CASL and is **module-scoped**: a role grants an action (`read`, `create`, `update`, `delete`) on a subject (`Account`, `Schedule`, `Skill`, `RightSizing`, …), and subjects map to product modules. + +Roles are assigned per tenant, so the same user can be an admin in one organisation and a viewer in another. + +## Tenant isolation + +Nucleus Ops is multi-tenant. All tenants share tables and are separated by a `tenant_id` column, enforced in the data-access layer rather than by convention: + +```typescript +const db = getTenantClient(tenantId); +await db.account.findMany(); // automatically scoped: WHERE tenant_id = tenantId +``` + + +`$executeRaw` bypasses the tenant extension. Raw SQL must add `WHERE tenant_id = $1` +manually. If you are auditing this codebase or contributing to it, raw SQL is the first +place to look. + + +Agent memory is deliberately **shared within a tenant** — a lesson learned by one user is available to their teammates. It is never shared across tenants. + +## Agent actions + +The AI agent can call tools that change real infrastructure, so it is gated: + +- **Approval by default.** Mutating tool calls pause and wait for a human decision. Auto-approve exists but is opt-in, per session. +- **Everything is audited.** Any action that touches an AWS resource is written to the audit log with the actor, target, and outcome. +- **Tool access is filtered.** Only enabled Skills, MCP servers, and knowledge bases are reachable at runtime — disabling one removes it from the agent's options, not just from the picker. +- **Autonomous skills need a human.** Skills synthesised from learned rules are created read-only and require explicit approval; disabling a skill acts as a veto. + +## Encryption and secrets + +- RDS PostgreSQL is encrypted at rest with AWS KMS; S3 buckets use server-side encryption. +- Application secrets live in AWS Secrets Manager; Pulumi state is encrypted with a KMS key. +- Provider credentials stored by the platform (for example LLM provider keys) are encrypted with AES-GCM before being written. +- Secrets shown in settings are masked by default; revealing one requires the right permission and writes an audit entry. + +## Auditing + +The audit log is append-only and records user and system actions with severity, actor, target resource, and result. It supports filtering and CSV export for compliance reviews. + +## Reporting a vulnerability + +Please **do not** open a public GitHub issue. Report privately via GitHub Security Advisories or email — see [SECURITY.md](https://github.com/kartikmanimuthu/nucleus-cloud-ops/blob/master-v1/SECURITY.md) for the process and response times. diff --git a/docs/AI_OPS_ARCHITECTURE.md b/docs/AI_OPS_ARCHITECTURE.md index 96b23c51..95a6cc8d 100644 --- a/docs/AI_OPS_ARCHITECTURE.md +++ b/docs/AI_OPS_ARCHITECTURE.md @@ -128,17 +128,17 @@ flowchart TB ### AI & Data | Technology | Purpose | |------------|---------| -| AWS Bedrock | LLM Provider (Claude Sonnet for logic, Amazon Titan for Embeddings) | -| DynamoDB | Real-time state (LangGraph checkpoints, App data, RBAC, Single Table Design) | -| Amazon S3 Tables | Apache Iceberg formatted Data Lake for multi-account resource inventory | -| cdk-s3-vectors | Vector embeddings store for RAG and semantic search operations | +| AWS Bedrock | LLM Provider (Claude Sonnet 4.6 for reasoning, Amazon Titan for embeddings) | +| PostgreSQL (RDS) | All persistent state — app data, RBAC, inventory, audit, LangGraph checkpoints, chat history | +| pgvector | Vector embeddings (HNSW index) for RAG, semantic search, and long-term agent memory | +| Amazon S3 | Blob storage — agent scratch space, knowledge-base sources, large checkpoint offload | ### Infrastructure | Technology | Purpose | |------------|---------| -| AWS CDK | Infrastructure as Code | -| AWS ECS Fargate | Serverless execution environment for Next.js and LangGraph Agent | -| AWS Lambda | Event-driven processors (Vector embeddings generation, Schedulers) | +| Pulumi | Infrastructure as Code (two stacks: `infra/networking` then `infra/compute`) | +| AWS ECS Fargate | Runs both services — `web-ui` (Next.js + LangGraph agents) and `workers` | +| pg-boss | Background job runner inside the `workers` service (no AWS Lambda is used) | | AWS Cognito | User Identity & Authentication | ## AI Ops Agent Workflow diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 745c2313..514bf446 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -53,14 +53,15 @@ flowchart TB Titan["Amazon Titan
Embeddings v2"]:::aws end - %% Data Store Tier (DynamoDB) + %% Data Store Tier (PostgreSQL + pgvector) subgraph Data_Stores - AppTable[(App Table)]:::database - UsersTable[(Users and Teams)]:::database + AppTable[(App Data)]:::database + UsersTable[(Users, Tenants and RBAC)]:::database AuditTable[(Audit Logs)]:::database LGCheckpoints[(LangGraph
Checkpoints)]:::database - LGConversations[(Agent
Conversations)]:::database - InventoryDDB[(Inventory Table)]:::database + LGConversations[(Agent Chat
History)]:::database + InventoryDB[(Inventory)]:::database + VectorStore[(pgvector
Embeddings)]:::database end %% Storage & Analytics Tier (S3 & S3 Tables/Iceberg) @@ -138,17 +139,17 @@ flowchart TB ### AI & Data | Technology | Purpose | |------------|---------| -| AWS Bedrock | LLM Provider (Claude Sonnet for logic, Amazon Titan for Embeddings) | -| DynamoDB | Real-time state (LangGraph checkpoints, App data, RBAC, Single Table Design) | -| Amazon S3 Tables | Apache Iceberg formatted Data Lake for multi-account resource inventory | -| cdk-s3-vectors | Vector embeddings store for RAG and semantic search operations | +| AWS Bedrock | LLM Provider (Claude Sonnet 4.6 for reasoning, Amazon Titan for embeddings) | +| PostgreSQL (RDS) | All persistent state — app data, RBAC, inventory, audit, LangGraph checkpoints, chat history | +| pgvector | Vector embeddings (HNSW index) for RAG, semantic search, and long-term agent memory | +| Amazon S3 | Blob storage — agent scratch space, knowledge-base sources, large checkpoint offload | ### Infrastructure | Technology | Purpose | |------------|---------| -| AWS CDK | Infrastructure as Code | -| AWS ECS Fargate | Serverless execution environment for Next.js and LangGraph Agent | -| AWS Lambda | Event-driven processors (Vector embeddings generation, Schedulers) | +| Pulumi | Infrastructure as Code (two stacks: `infra/networking` then `infra/compute`) | +| AWS ECS Fargate | Runs both services — `web-ui` (Next.js + LangGraph agents) and `workers` | +| pg-boss | Background job runner inside the `workers` service (no AWS Lambda is used) | | AWS Cognito | User Identity & Authentication | ## AI Ops Agent Workflow diff --git a/docs/IMPLEMENTATION_MULTIMODAL.md b/docs/IMPLEMENTATION_MULTIMODAL.md index 613072ef..fd165bf3 100644 --- a/docs/IMPLEMENTATION_MULTIMODAL.md +++ b/docs/IMPLEMENTATION_MULTIMODAL.md @@ -126,11 +126,11 @@ new HumanMessage({ ## Supported Models All models in the platform support multimodal: -- ✅ Claude 4.5 Sonnet -- ✅ Claude 4.5 Haiku -- ✅ Claude 4.5 Opus -- ✅ Claude 4.6 Sonnet -- ✅ Claude 4.6 Opus +- ✅ Claude Sonnet 4.5 +- ✅ Claude Haiku 4.5 +- ✅ Claude Opus 4.5 +- ✅ Claude Sonnet 4.6 +- ✅ Claude Opus 4.6 - ✅ Nova 2 Lite ## Agent Compatibility @@ -193,7 +193,7 @@ Covers: ### Token Usage - Images increase token consumption significantly -- Claude 4.5 Sonnet: ~1,600 tokens per 1024x1024 image +- Claude Sonnet 4.5: ~1,600 tokens per 1024x1024 image - Larger/more complex images = more tokens ### Response Time diff --git a/docs/MULTIMODAL_ARCHITECTURE.md b/docs/MULTIMODAL_ARCHITECTURE.md index 4a474027..9d507319 100644 --- a/docs/MULTIMODAL_ARCHITECTURE.md +++ b/docs/MULTIMODAL_ARCHITECTURE.md @@ -121,7 +121,7 @@ ┌─────────────────────────────────────────────────────────────────────┐ │ AWS BEDROCK │ │ ┌────────────────────────────────────────────────────────────┐ │ -│ │ Claude 4.5 Sonnet (Vision Model) │ │ +│ │ Claude Sonnet 4.5 (Vision Model) │ │ │ │ │ │ │ │ 1. Processes text prompt │ │ │ │ 2. Analyzes image content │ │ diff --git a/docs/MULTIMODAL_SUPPORT.md b/docs/MULTIMODAL_SUPPORT.md index 8fa1a4d8..a728f642 100644 --- a/docs/MULTIMODAL_SUPPORT.md +++ b/docs/MULTIMODAL_SUPPORT.md @@ -15,11 +15,11 @@ Nucleus Cloud Ops now supports multimodal interactions, allowing users to upload ## Supported Models All Claude models in the platform support multimodal inputs: -- Claude 4.5 Sonnet -- Claude 4.5 Haiku -- Claude 4.5 Opus -- Claude 4.6 Sonnet -- Claude 4.6 Opus +- Claude Sonnet 4.5 +- Claude Haiku 4.5 +- Claude Opus 4.5 +- Claude Sonnet 4.6 +- Claude Opus 4.6 Amazon Nova models also support multimodal: - Nova 2 Lite diff --git a/docs/schema-design.md b/docs/schema-design.md index 682bce9e..68fb736d 100644 --- a/docs/schema-design.md +++ b/docs/schema-design.md @@ -1,5 +1,9 @@ # DynamoDB Schema Design - Nucleus Platform +> **⚠️ Historical document.** This describes the original DynamoDB single-table design, +> which has been **fully replaced by PostgreSQL + Prisma**. It is kept for context on the +> migration only. The current data model is `libs/prisma/schema.prisma`. + ## 1. Overview The platform utilizes two distinct DynamoDB tables: 1. **NucleusAppTable**: Stores configuration logic (Accounts, Schedules, Mappings) using Single Table Design. diff --git a/docs/techstack.md b/docs/techstack.md index b1240cea..977fb895 100644 --- a/docs/techstack.md +++ b/docs/techstack.md @@ -1,4 +1,4 @@ -# Nucleus Cloud Ops — Tech Stack +# Nucleus Ops — Tech Stack A multi-tenant **AWS Cloud Operations Platform**: multi-account resource scheduling + an AI Ops agent powered by AWS Bedrock. This document catalogs the full stack from a full-stack development standpoint, sourced from the live `package.json` manifests and `CLAUDE.md`. @@ -10,9 +10,9 @@ A multi-tenant **AWS Cloud Operations Platform**: multi-account resource schedul |---|---| | Monorepo orchestrator | **Nx 21** (bare `nx` invoked from root npm scripts) | | Package manager / runtime | **Bun 1.2.12** (workspaces, single hoisted install) | -| Node baseline | **Node.js 20.x** (`>=20.0.0`; Lambda requires 20+) | -| Language | **TypeScript ~5.6.2** (infra/root) · `^5.7.2` (workers) · `^5` (web-ui) · `^5.0.0` (lambdas) | -| Build bundlers | esbuild (scheduler Lambda), tsc (workers, kb_sync Lambda), Next webpack (web-ui), Pulumi (infra) | +| Node baseline | **Node.js 20.x** (`>=20.0.0`) | +| Language | **TypeScript ~5.6.2** (infra/root) · `^5.7.2` (workers) · `^5` (web-ui) | +| Build bundlers | tsc (workers), Next webpack (web-ui), Pulumi (infra) | | Task visualization | `nx graph` | --- @@ -27,7 +27,7 @@ A multi-tenant **AWS Cloud Operations Platform**: multi-account resource schedul | Component primitives | **Radix UI** (≈27 primitives: dialog, dropdown-menu, select, tabs, toast, tooltip, accordion, etc.) in shadcn/ui style; `class-variance-authority`, `clsx`, `tailwind-merge` (`cn()`), `cmdk`, `vaul`, `sonner`, `input-otp`, `embla-carousel-react`, `react-resizable-panels` | | Icons | **lucide-react** ^0.454.0 | | Tables | **TanStack React Table** ^8.21.3 | -| Forms/validation | **react-hook-form** ^7.54.1 + `@hookform/resolvers` + **Zod** ^3.24.1 | +| Forms/validation | **react-hook-form** ^7.54.1 + `@hookform/resolvers` + **Zod** ^4.0.0 | | Charts | **Recharts** (latest) | | Code editor | **Monaco** (`@monaco-editor/react` ^4.7.0) | | Docs/MDX | **fumadocs** (`fumadocs-core` ^14.7.7, `fumadocs-ui` ^14.7.7, `fumadocs-mdx` ^10.0.2) + `@mdx-js/react`, `remark-gfm`, `react-markdown` | @@ -71,7 +71,7 @@ A multi-tenant **AWS Cloud Operations Platform**: multi-account resource schedul |---|---| | Agent orchestration | **LangGraph** ^1.2.0 + **LangChain** ^1.2.28 (`@langchain/core` ^1.1.39, `@langchain/aws` ^1.3.0, `@langchain/openai` ^1.3.0) | | Streaming hooks (UI) | **Vercel AI SDK** (`ai` ^5.0.115, `@ai-sdk/react`, `@ai-sdk/amazon-bedrock`, `@ai-sdk/anthropic`, `@ai-sdk/langchain`) | -| LLM provider | **AWS Bedrock** — Claude 4.5 Sonnet | +| LLM provider | **AWS Bedrock** — Claude Sonnet 4.6 (`anthropic.claude-sonnet-4-6`) | | MCP integration | `@modelcontextprotocol/sdk` ^1.26.0 | | Deep agent framework | `deepagents` ^1.8.1 | | Observability | **Langfuse** (`langfuse-langchain` ^3.38.6) | @@ -84,7 +84,7 @@ A multi-tenant **AWS Cloud Operations Platform**: multi-account resource schedul | Concern | Choice | |---|---| | Worker process | Single Node process via **pg-boss** (`pg-boss` ^10.1.5) | -| Jobs | scheduler/, discovery/, kb-sync/ | +| Jobs | scheduler/, discovery/, right-sizing/, kb-sync/, agent-ops-scheduler/, certificate-expiry-monitor/ | | AWS SDK | **AWS SDK v3** — ~30 service clients across web-ui + workers (EC2, ECS, RDS, S3, SQS, STS, EventBridge, CloudWatch, IAM, Cognito, Lambda, ACM, WAFv2, EKS, EFS, Elasticache, CodePipeline, Secrets Manager, SSM, Backup, etc.) | | Cross-account | STS `AssumeRole` exclusively (never hardcoded creds) | | Concurrency | `p-limit` ^7.3.0 | @@ -93,14 +93,20 @@ A multi-tenant **AWS Cloud Operations Platform**: multi-account resource schedul --- -## AWS Lambda Functions (`lambda/`) +## Background Jobs (`apps/workers/src/jobs/`) -| Function | Language | Build | Tests | -|---|---|---|---| -| `scheduler/` | TypeScript ^5.7.2 | esbuild | Vitest ^2.1.8 | -| `discovery/` | Python 3.12 (boto3, pyiceberg, pyarrow, pandas) | — | — | -| `vector_processor/` | Python + TypeScript | — | — | -| `kb_sync_processor/` | TypeScript ^5.0.0 | tsc | — | +There are **no AWS Lambda functions** in this repository. The former Lambdas were migrated +to pg-boss jobs that run inside the `workers` ECS service, and the Python discovery +function was rewritten in TypeScript. + +| Job | Purpose | +|---|---| +| `scheduler/` | Evaluates schedules and starts/stops resources across accounts | +| `discovery/` | Multi-account, multi-region resource scan (AWS SDK v3) | +| `right-sizing/` | CloudWatch-based recommendations + weekly pricing refresh | +| `kb-sync/` | Knowledge-base ingestion and embedding (S3, Bitbucket, Confluence) | +| `agent-ops-scheduler/` | Fires scheduled autonomous agent runs | +| `certificate-expiry-monitor/` | Flags ACM certificates nearing expiry | --- @@ -109,7 +115,7 @@ A multi-tenant **AWS Cloud Operations Platform**: multi-account resource schedul | Stack | Manages | Libs | |---|---|---| | `infra/networking` | VPC, subnets, subnet groups | `@pulumi/pulumi` ^3.228, `@pulumi/aws` ^7.23, `@pulumi/awsx` ^3.3.1 | -| `infra/compute` | ECS Fargate, RDS PostgreSQL, Cognito, CloudFront, Lambda, SQS, EventBridge, S3 | `@pulumi/pulumi`, `@pulumi/aws`, `@pulumi/awsx` ^3.4, `@pulumi/command` ^1.2.1, `@pulumi/random` ^4.19.1 | +| `infra/compute` | ECS Fargate (web-ui + workers), RDS PostgreSQL, Cognito, CloudFront, S3 (no Lambda) | `@pulumi/pulumi`, `@pulumi/aws`, `@pulumi/awsx` ^3.4, `@pulumi/command` ^1.2.1, `@pulumi/random` ^4.19.1 | Deploy order: **networking → compute**. State backend: S3 (`s3://nucleus-pulumi-state`), secrets via KMS (`awskms://alias/pulumi-secrets`). Prod profile: `PLATFORM-ADMIN`. @@ -121,7 +127,6 @@ Deploy order: **networking → compute**. State backend: S3 (`s3://nucleus-pulum |---|---| | Unit — web-ui | **Vitest** ^4.0.18 + `@vitest/coverage-v8` | | Unit — workers | **Vitest** ^2.1.8 | -| Unit — scheduler Lambda | Vitest ^2.1.8 (own setup) | | Unit — root | Jest ^29.7.0 + ts-jest ^29.2.5 | | Property-based | `fast-check` ^4.5.3 | | E2E browser | **Playwright** ^1.58.2 (`apps/web-ui-e2e/`, `implicitDependencies: ["web-ui"]`, `webServer` auto-starts `bun run dev` on :3001) | @@ -141,10 +146,10 @@ Deploy order: **networking → compute**. State backend: S3 (`s3://nucleus-pulum | Concern | Choice | |---|---| -| Container runtime | **ECS Fargate** (web-ui on `node:20.9.0-slim` + AWS Lambda Web Adapter 0.8.4) | +| Container runtime | **ECS Fargate** — two services: `web-ui` and `workers`, both on `node:20-slim` | | Image build | Dockerfile installs with Bun, builds under `node:20-slim` via `npm run build` (real Node + fresh `.next`) | | CDN | **AWS CloudFront** in front of ALB + S3 — https://d11lr8aqp8vqde.cloudfront.net | -| Compute Lambdas | scheduler (EventBridge cron every 30 min), discovery, vector_processor, kb_sync_processor | +| Background jobs | pg-boss jobs in the `workers` service (per-tenant cron); no AWS Lambda | | Audit | Every AWS-mutating action logged via `AuditService` (`audit_log` table, 30-day TTL) | --- diff --git a/infra/cicd/README.md b/infra/cicd/README.md index 06232bb6..040b6ec8 100644 --- a/infra/cicd/README.md +++ b/infra/cicd/README.md @@ -1,4 +1,4 @@ -# Nucleus Cloud Ops — CI/CD Pipeline +# Nucleus Ops — CI/CD Pipeline AWS CodePipeline that builds, previews, and deploys the Pulumi infrastructure on every push to `master-v1`. diff --git a/infra/cicd/package.json b/infra/cicd/package.json index 51b81ce6..255b04e3 100644 --- a/infra/cicd/package.json +++ b/infra/cicd/package.json @@ -1,7 +1,7 @@ { "name": "nucleus-cicd", "version": "1.0.0", - "description": "Nucleus Cloud Ops — CodePipeline CI/CD (Pulumi)", + "description": "Nucleus Ops — CodePipeline CI/CD (Pulumi)", "main": "index.ts", "scripts": { "preview": "pulumi preview", diff --git a/infra/compute/package.json b/infra/compute/package.json index 2a75e1e3..1e22eeb6 100644 --- a/infra/compute/package.json +++ b/infra/compute/package.json @@ -1,7 +1,7 @@ { "name": "nucleus-compute", "version": "1.0.0", - "description": "Nucleus Cloud Ops — ECS, Lambda, DynamoDB, Cognito, CloudFront (Pulumi)", + "description": "Nucleus Ops — ECS, RDS PostgreSQL, Cognito, CloudFront (Pulumi)", "main": "index.ts", "scripts": { "preview": "pulumi preview", diff --git a/infra/networking/package.json b/infra/networking/package.json index 98321f23..3019b2ef 100644 --- a/infra/networking/package.json +++ b/infra/networking/package.json @@ -1,7 +1,7 @@ { "name": "nucleus-networking", "version": "1.0.0", - "description": "Nucleus Cloud Ops — VPC and networking (Pulumi)", + "description": "Nucleus Ops — VPC and networking (Pulumi)", "main": "index.ts", "scripts": { "preview": "pulumi preview",