Backend Lead Developer

About the Company

Web3 Index is a business development tool designed to discover, analyze, and curate the most complete information about blockchain and crypto projects. The platform aggregates and enriches data from sources like CoinGecko, CoinMarketCap, CypherHunter, and more. Using AI (ChatGPT, Perplexity, CrewAI), advanced scraping methods, and databases (PostgreSQL, MongoDB), Web3 Index delivers high-quality, useful ideas to users.

The platform powers:

• Data aggregation & enrichment (socials, team, LinkedIn, project metadata).
• AI-powered search & embeddings using MongoDB Vector DB.
• AI agents that research and validate project data.
• Integrations with TalkJS, Apollo, Slack, Twitter, Cloudflare, Postmark, Tranzilla, SimilarWeb, and more.

Responsibilities

• Lead backend development: Drive feature design, architecture decisions, and the technical plan for the platform.
• Research & Development (R&D): Experiment with new technologies, AI models, and scraping methods to continuously make data quality and system performance better.
• Web scraping & data pipelines: Design and maintain strong scraping systems using Python libraries (e.g., Scrapy, Requests, BeautifulSoup, Playwright).
• Database management: Adjust and maintain PostgreSQL (with Alembic/Alchemy) and MongoDB (vector search/embeddings).
• AI & agent integration: Work with LLMs (ChatGPT, Perplexity) and CrewAI to build and scale AI-driven features.
• DevOps & infrastructure: Manage AWS services (S3, EC2, Lambda, CloudWatch, API Gateway, SageMaker) and automation (Bash scripting, cron jobs).
• Blockchain expertise: Analyze and interact with smart contracts (Solidity – entry level), understand blockchain data structures, and integrate relevant protocols.
• Team leadership: Mentor developers, perform code reviews, enforce recommended approaches, and make sure high-quality deliverables are met.
• Integration management: Oversee and expand integrations with 3rd-party APIs and services (TalkJS, Slack, Twitter, Apollo, etc.).

Requirements

• Five or more years of backend development experience, with at least two years in a lead/architect role.
• Strong Python skills (FastAPI, Flask, Django or similar).
• Proven experience with web scraping at scale.
• Solid understanding of PostgreSQL (schemas, migrations, performance tuning) and MongoDB (vector DB, embeddings).
• Familiarity with AI/LLM frameworks and data pipelines.
• Hands-on AWS experience (EC2, S3, CloudWatch, API Gateway, Lambda, SageMaker).
• Knowledge of blockchain/web3 ecosystems, with the ability to read/write Solidity contracts (entry-level).
• Strong DevOps skills (Bash, cron, deployment automation).
• Excellent problem-solving skills and ability to take ownership of complex systems.
• Strong communication and leadership abilities.

Preferred Qualifications (Nice to Have)

• Experience with CrewAI or other agent frameworks.
• Knowledge of data enrichment and NLP pipelines.
• Exposure to business intelligence tools and analytics.
• Open-source contributions or published work in scraping/AI/blockchain.

What We Offer

• Opportunity to lead a modern Web3/AI product.
• Work with modern stacks (AI, blockchain, web scraping, embeddings).
• Flexible remote-first work environment.
• Growth potential in leadership and architecture.
• Competitive compensation package.