Backend
Integration of NLP Model with Flask Backend
This section documents the transition of the NLP model from a CLI-based interface to a web-integrated backend architecture using Flask. The modifications allow the entire model pipeline to function via HTTP requests, improving accessibility, automation, and frontend interaction. The model.py file is adapted for integration with the Flask backend (app.py) instead of using a command-line interface. The process_query() function is designed to take structured inputs (coin, query) and return structured JSON outputs for rendering in the frontend.
The function `process_query(coin, query)` is the main bridge between the Flask backend and the NLP model. It takes a cryptocurrency name and user input as arguments and returns structured results suitable for JSON serialization and frontend display.
This function: - Analyzes sentiment using TextBlob - Retrieves top matching answers using BM25 and TF-IDF algorithms - Combines results and sorts them - Returns a dictionary with formatted output and raw data for logging
The `raw_top_results` field is retained only for server-side logging purposes. It is removed before the frontend receives the data. This keeps logs comprehensive while maintaining frontend efficiency.
The `raw_top_results` field is retained only for server-side logging purposes. It is removed before the frontend receives the data. This keeps logs comprehensive while maintaining frontend efficiency.
Query Logging via logger.py
Each processed query is logged into `query_logs.json` for tracking. The log entry includes: - Timestamp of query - Coin name - User question - Detected sentiment - Top answer text and scores This is handled by the `log_query()` function, which appends logs in a JSON array format using Python's built-in `json` module.
Summing up Key Enhancements:
Unified Query Handler for API:
def process_query(coin, query):
Replaces CLI interaction with a reusable backend-compatible function.
Accepts two parameters: the selected cryptocurrency (coin) and user question (query).
Returns a structured response dictionary that includes:
Original query
Coin name
Sentiment result (with score)
Ranked top 3 answers (as formatted strings)
Raw answer-score pairs for logging
Why this matters: Previously, the output was printed to the terminal. Now, it is returned to the Flask route /query where it can be serialized to JSON and sent to the frontend.
Output Formatting for Frontend Rendering:
Ensures each result is trimmed and human-readable.
Designed for HTML rendering inside the web interface (index.html).
3. Compatibility with Logging Module
This line is included in the returned dictionary only for backend logging.
It is removed before sending the response to the frontend in app.py.
Coin Validation Integration
Now shared across model.py and app.py.
Ensures any coin selected in the frontend is validated before query processing.
Removed Command-Line Interface
The interactive loop when writing code for model (asking the user to type the coin and query) has been removed.
Replaced with stateless processing suited for HTTP requests.
# query_logs.json
Purpose: It is used to store a history of user queries and system responses during the text analysis process.
Usage in code:
Error Tracking: Can log query failures (e.g., empty results or low BM25 scores).
Audit Trail: Used to validate if the system is returning accurate and relevant answers over time.
Key Features:
JSON Format: Stores logs in a structured JSON array for easy access and readability.
Log Entries: Each log typically contains a timestamp, query text, response, sentiment, and confidence score.
Importance:
Transparency: Enables retrospective review of system outputs.
Model Improvement: Helps analyze mismatches between query intent and retrieved sentences.\
User Insight: Useful for visualizing user interest trends over time.
Testing & Debugging: Makes regression testing easier by replaying previous queries.
Purpose: It is used to store a history of user queries and system responses during the text analysis process.
Usage in code:
Error Tracking: Can log query failures (e.g., empty results or low BM25 scores).
Audit Trail: Used to validate if the system is returning accurate and relevant answers over time.
Key Features:
JSON Format: Stores logs in a structured JSON array for easy access and readability.
Log Entries: Each log typically contains a timestamp, query text, response, sentiment, and confidence score.
Importance:
Transparency: Enables retrospective review of system output.
Model Improvement: Helps analyze mismatches between query intent and retrieved sentences.
User Insight: Useful for visualizing user interest trends over time.
Testing & Debugging: Makes regression testing easier by replaying previous queries.
# Keywords.py
Purpose: It is responsible for extracting or managing key cryptocurrency-related terms or phrases from articles. These keywords are used to enhance search relevance, preserve domain-specific context during preprocessing, and improve user query matching in the BM25 retrieval step.
Key Features:
Crypto-Specific Vocabulary: Maintains a curated list of crypto terms (e.g., "blockchain", "DeFi", "halving", "altcoin").
Custom Stopword Exceptions: Prevents removal of important crypto terms during preprocessing (e.g., during stopword filtering).
Keyword Matching: May provide utility functions to identify or highlight keyword presence in text.
Query Expansion (if implemented): Can expand user queries with related keywords for broader match.
Usage in code:
Preprocessing (Step: preprocess_sentence)
Ensures that key crypto terms are not stemmed or lemmatized to preserve their semantic identity.
Prevents accidental removal during stopword or special character cleaning.
Sentiment or Relevance Analysis:
Used to filter or tag sentences that contain crypto-related terms for priority scoring.
BM25 Index Creation:
Can be used to label which sentences are “keyword-rich” to weigh them higher in ranking.
Importance:
Domain Sensitivity: Crypto-specific terms often don't behave like general English (e.g., “hodl” isn’t in any dictionary). Preserving these terms improves model accuracy.
Search Optimization: Ensures BM25 index or query expansion functions operate with context-aware terms.
User Intent Matching: Improves alignment between user queries and text content by anchoring on domain keywords.
Result of keywords.py in a separate json file for future QA:
# Logger.py
Purpose: It is used to configure and manage logging across the entire crypto text analysis pipeline. It centralizes the setup of the logging system so that other modules can easily import and use a consistent logging format and level. This helps in monitoring, debugging, and auditing the system.
Key Features:
Centralized Logger Configuration: Defines logging settings (level, format, output file).
Reusable Across Modules: Any script (e.g., scraper.py, bm25.py) can import the configured logger and write logs.
File and Console Output: Optionally writes logs to both the terminal and a log file (e.g., pipeline.log).
Custom Format: Includes timestamps, log levels (INFO, WARNING, ERROR), and message content.
Usage in code:
Imported into modules like scraper.py or bm25.py to log progress updates (e.g., "Scraping Bitcoin - Page 1"), warnings (e.g., missing articles), and errors (e.g., failed network requests) using logger.info(), logger.warning(), and logger.error().
Ensures all log messages are consistently formatted and saved to both the console and a log file (pipeline.log), supporting easy debugging and execution tracking.
Importance:
Traceability: Tracks every step (e.g., "Preprocessing complete", "BM25 index created").
Debugging: Helps locate where and why failures occurred (e.g., network issues, parsing errors).
Monitoring Progress: Useful for long-running processes like scraping multiple pages.
Maintenance: Provides system-level visibility for developers working on different modules.
Purpose: It serves as the main entry point for the application, integrating all components of the text analysis pipeline. It handles user input, processes queries using the BM25 model, and returns relevant answers with associated sentiment. It also initializes required resources such as the cleaned data, sentiment scores, and BM25 index.
Key Features:
User Interaction: Accepts user queries (via terminal or web interface) and returns ranked answers.
BM25 Integration: Leverages the BM25 model to retrieve the most relevant sentences from the processed dataset.
Sentiment Display: Enhances answers with sentiment scores and categories (positive, negative, neutral).
Modular Calls: Ties together preprocessed text, BM25 search, and sentiment data from other modules.
Execution Control: Acts as the main script run to launch the full pipeline or demo the system.
Usage in code:
Executes BM25 search to retrieve and rank the top N relevant sentences.
Displays matched sentences along with metadata like BM25 score, sentiment polarity, and article source.
Optionally logs the query and results into query_logs.json for tracking and debugging.
Importance:
Integration Point: Bridges data preprocessing, keyword relevance, and sentiment analysis into one cohesive flow.
User Interface Layer: Acts as the main access point for users to interact with the system.
Testing & Deployment: Ideal for quickly testing the pipeline with different queries or deploying a minimal prototype.
# Keywords.py
Purpose:
It is responsible for extracting or managing key cryptocurrency-related terms or phrases from articles. These keywords are used to enhance search relevance, preserve domain-specific context during preprocessing, and improve user query matching in the BM25 retrieval step.
Key Features:
Crypto-Specific Vocabulary: Maintains a curated list of crypto terms (e.g., "blockchain", "DeFi", "halving", "altcoin").
Custom Stopword Exceptions: Prevents removal of important crypto terms during preprocessing (e.g., during stopword filtering).
Keyword Matching: May provide utility functions to identify or highlight keyword presence in text.
Query Expansion (if implemented): Can expand user queries with related keywords for broader match.
Usage in code:
Preprocessing (Step: preprocess_sentence)
Ensures that key crypto terms are not stemmed or lemmatized to preserve their semantic identity.
Prevents accidental removal during stopword or special character cleaning.
Sentiment or Relevance Analysis:
Used to filter or tag sentences that contain crypto-related terms for priority scoring.
BM25 Index Creation:
Can be used to label which sentences are “keyword-rich” to weigh them higher in ranking.
Importance:
Domain Sensitivity: Crypto-specific terms often don't behave like general English (e.g., “hodl” isn’t in any dictionary). Preserving these terms improves model accuracy.
Search Optimization: Ensures BM25 index or query expansion functions operate with context-aware terms.
User Intent Matching: Improves alignment between user queries and text content by anchoring on domain keywords.
app.py (continued)
Libraries Used:
/ – serves the frontend index.html
/query – accepts JSON input (coin + query) and returns results
Endpoint Logic Breakdown:
/ Route (GET)
Result of keywords.py in a separate json file for future QA:
# Logger.py
Purpose:
It is used to configure and manage logging across the entire crypto text analysis pipeline. It centralizes the setup of the logging system so that other modules can easily import and use a consistent logging format and level. This helps in monitoring, debugging, and auditing the system.
Key Features:
Centralized Logger Configuration: Defines logging settings (level, format, output file).
Reusable Across Modules: Any script (e.g., scraper.py, bm25.py) can import the configured logger and write logs.
File and Console Output: Optionally writes logs to both the terminal and a log file (e.g., pipeline.log).
Custom Format: Includes timestamps, log levels (INFO, WARNING, ERROR), and message content.
Usage in code:
Imported into modules like scraper.py or bm25.py to log progress updates (e.g., "Scraping Bitcoin - Page 1"), warnings (e.g., missing articles), and errors (e.g., failed network requests) using logger.info(), logger.warning(), and logger.error().
Ensures all log messages are consistently formatted and saved to both the console and a log file (pipeline.log), supporting easy debugging and execution tracking.
Importance:
Traceability: Tracks every step (e.g., "Preprocessing complete", "BM25 index created").
Debugging: Helps locate where and why failures occurred (e.g., network issues, parsing errors).
Monitoring Progress: Useful for long-running processes like scraping multiple pages.
Maintenance: Provides system-level visibility for developers working on different modules.
Purpose:
It serves as the main entry point for the application, integrating all components of the text analysis pipeline. It handles user input, processes queries using the BM25 model, and returns relevant answers with associated sentiment. It also initializes required resources such as the cleaned data, sentiment scores, and BM25 index.
Key Features:
User Interaction: Accepts user queries (via terminal or web interface) and returns ranked answers.
BM25 Integration: Leverages the BM25 model to retrieve the most relevant sentences from the processed dataset.
Sentiment Display: Enhances answers with sentiment scores and categories (positive, negative, neutral).
Modular Calls: Ties together preprocessed text, BM25 search, and sentiment data from other modules.
Execution Control: Acts as the main script run to launch the full pipeline or demo the system.
Usage in code:
Executes BM25 search to retrieve and rank the top N relevant sentences.
Displays matched sentences along with metadata like BM25 score, sentiment polarity, and article source.
Optionally logs the query and results into query_logs.json for tracking and debugging.
Importance:
Integration Point: Bridges data preprocessing, keyword relevance, and sentiment analysis into one cohesive flow.
User Interface Layer: Acts as the main access point for users to interact with the system.
Testing & Deployment: Ideal for quickly testing the pipeline with different queries or deploying a minimal prototype.
app.py (continued)
Libraries Used:
/ – serves the frontend index.html
/query – accepts JSON input (coin + query) and returns results
Endpoint Logic Breakdown:
/ Route (GET)
Loads the interface via index.html from the templates/ directory.
Fallbacks to a JSON message in case of errors (e.g., missing template).
/query Route (POST)
Handles the backend logic:
Input Handling and Validation:
Validates presence of data from request body.
Coin and Query Validation:
Strips leading/trailing spaces
Normalizes to lowercase
Validates against supported coins defined in model.py
Ensures query relevance using keywords (valid_query(query))
Query Processing:
Calls core logic from model.py
Returns insights including sentiment and top-ranked responses
Logging User Interaction:
Delegates logging to logger.py for persistent audit trail.
Output:
Returns formatted response to frontend.
logger.py
Purpose: Logs each interaction into a local query_logs.json file with:
Timestamp
Coin name
User’s query
Sentiment result
Top answers with scores
Libraries Used:
Key Function: def log_query(coin, user_query, sentiment, top_results):
Creates an entry object
Appends to existing log file or creates a new one
Maintains logs in list format
Resilience: Wrapped in a try-except block to handle logging errors without affecting backend flow.
Loads the interface via index.html from the templates/ directory.
Fallbacks to a JSON message in case of errors (e.g., missing template).
/query Route (POST)
Handles the backend logic:
Input Handling and Validation:
Validates presence of data from request body.
Coin and Query Validation:
Strips leading/trailing spaces
Normalizes to lowercase
Validates against supported coins defined in model.py
Ensures query relevance using keywords (valid_query(query))
Query Processing:
Calls core logic from model.py
Returns insights including sentiment and top-ranked responses
Logging User Interaction:
Delegates logging to logger.py for persistent audit trail.
Output:
Returns formatted response to frontend.
logger.py
Purpose: Logs each interaction into a local query_logs.json file with:
Timestamp
Coin name
User’s query
Sentiment result
Top answers with scores
Libraries Used:
Key Function:
def log_query(coin, user_query, sentiment, top_results):
Creates an entry object
Appends to existing log file or creates a new one
Maintains logs in list format
Resilience: Wrapped in a try-except block to handle logging errors without affecting backend flow.
Last updated