Acceptable Use Policy

1. Purpose & Ethical Foundation

This Acceptable Use Policy (“AUP”) defines the mandatory standards of conduct governing all customers, users, and automated tasks utilizing Crawlexa extraction clusters, scraping APIs, residential proxy pools, and managed pipelines.

Crawlexa was founded on the principle that the open web fosters fair market competition and price transparency. However, automated collection must always be conducted responsibly, respectfully, and in strict compliance with applicable privacy, cybersecurity, and consumer protection laws.

2. Permitted Extraction Activities

Crawlexa services may be engaged strictly for the collection and processing of publicly accessible, non-confidential, and non-PII information. Standard permitted enterprise use cases include:

• eCommerce & Retail Catalog Intelligence: Public prices, promotional discounts, buybox winners, stock availability, ratings, and public product specifications.
• Digital Shelf & Search Visibility: Keyword search placements, sponsored advertising rankings, and store fulfillment options across web and native mobile apps.
• Brand Protection & MAP Governance: Discovering unauthorized marketplace sellers, counterfeit listings, and minimum advertised price (MAP) compliance deviations.
• Market & Alternative Data Research: Aggregated job posting counts, real estate listing dynamics, airline fare schedules, and consumer review sentiment.
• AI & LLM Pre-Training: Sourcing clean, open-web text, public documentation, and research literature for machine learning models.

3. Strictly Prohibited Uses

Clients and API users may NOT utilize Crawlexa infrastructure to:

• Harvest Non-Public or Protected Data: Attempting to access data protected by authentication credentials, paywalls, password-protected portals, or private user dashboards.
• Scrape Personally Identifiable Information (PII): Collecting personal names, residential addresses, personal phone numbers, email directories, financial records, credit cards, or protected health information (PHI/HIPAA).
• Server Denial-of-Service or Infrastructure Overload: Executing aggressive request loops, unbounded concurrency, or traffic floods that impair or degrade the performance, uptime, or availability of target web hosts.
• Bypass Legitimate Security Controls with Malicious Intent: Exploiting application vulnerabilities, conducting penetration tests against unauthorized third parties, or executing injection attacks (SQLi, XSS, SSRF).
• Circumvent Intellectual Property Protections: Extracting proprietary source code, confidential databases, or copyrighted audio/video streams for illegal redistribution.
• Facilitate Harassment, Fraud, or Surveillance: Tracking private individuals, enabling unauthorized background checks, or scraping personal profiles for identity theft or predatory lending.

4. Crawler Hygiene & Host Respect Controls

To protect host server availability, all Crawlexa managed pipelines and API instances enforce autonomous guardrails:

• Token-Bucket Concurrency Limits: Requests per domain are bounded dynamically based on host latency and historical capacity.
• Automated Backoff: Immediate 30-to-300 second exponential backoff upon encountering HTTP 429 (Too Many Requests) or HTTP 503 (Service Unavailable) status codes.
• Respect for Private Robots Paths: Automated scrapers refrain from indexing private transactional areas (such as /admin/, /checkout/, /cart/).
• Automated PII Scrubbing: Pre-serialization filters automatically redact any accidentally captured personal contact strings.

5. Compliance Monitoring & Abuse Reporting

Crawlexa continuously monitors infrastructure telemetry for abnormal patterns. If you are a site operator, privacy officer, or security researcher and observe traffic from our IP network that violates this policy, please report it immediately:

Upon receipt of a verified abuse notification, Crawlexa site reliability engineers will immediately investigate, apply host-level rate limiting, or add the reported domain to our permanent exclusion list.