gavindiaz/solana-streamer
1
0
Fork 0
mirror of https://github.com/0xfnzero/solana-streamer.git synced 2026-08-20 20:38:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix channel is full

Browse Source
This commit is contained in:
wood
2025-08-04 21:30:55 +08:00
parent 109c50cf06
commit d68bdc1d76
6 changed files with 880 additions and 298 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+36 -146
View File
@@ -17,9 +17,14 @@ A lightweight Rust library for real-time event streaming from Solana DEX trading
5. **Unified Event Interface**: Consistent event handling across all supported protocols
6. **Event Parsing System**: Automatic parsing and categorization of protocol-specific events
7. **High Performance**: Optimized for low-latency event processing
8. **Batch Processing**: Efficient event batching to improve throughput and reduce overhead
9. **Performance Monitoring**: Built-in performance metrics and monitoring capabilities
10. **Memory Optimization**: Object pooling and caching to reduce memory allocations
8. **Batch Processing Optimization**: Batch processing events to reduce callback overhead
9. **Performance Monitoring**: Built-in performance metrics monitoring, including event processing speed, memory usage, etc.
10. **Memory Optimization**: Object pooling and caching mechanisms to reduce memory allocations
11. **Flexible Configuration System**: Support for custom batch sizes, backpressure strategies, channel sizes, and other parameters
12. **Preset Configurations**: Provides high-performance, low-latency, ordered processing, and other preset configurations
13. **Backpressure Handling**: Supports blocking, dropping, retrying, ordered, and other backpressure strategies
14. **Runtime Configuration Updates**: Supports dynamic configuration parameter updates at runtime
15. **Full Function Performance Monitoring**: All subscribe_events functions support performance monitoring, automatically collecting and reporting performance metrics
## Installation
@@ -48,7 +53,7 @@ solana-streamer-sdk = "0.1.9"
## Usage Examples
### Basic Usage with Performance Monitoring
### Advanced Usage with Batch Processing and Backpressure
```rust
use solana_streamer_sdk::{
@@ -89,11 +94,14 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
async fn test_grpc() -> Result<(), Box<dyn std::error::Error>> {
println!("Subscribing to Yellowstone gRPC events...");
// Create gRPC client with performance monitoring enabled
// Create low-latency configuration
let mut config = ClientConfig::low_latency();
// Enable performance monitoring, has performance overhead, disabled by default
config.enable_metrics = true;
let grpc = YellowstoneGrpc::new_with_config(
"https://solana-yellowstone-grpc.publicnode.com:443".to_string(),
None,
true, // enable performance monitoring
config,
)?;
let callback = create_event_callback();
@@ -109,11 +117,12 @@ async fn test_grpc() -> Result<(), Box<dyn std::error::Error>> {
// Configure account filtering
let account_include = vec![
PUMPFUN_PROGRAM_ID.to_string(), // Listen to pumpfun program ID
PUMPSWAP_PROGRAM_ID.to_string(), // Listen to pumpswap program ID
BONK_PROGRAM_ID.to_string(), // Listen to bonk program ID
RAYDIUM_CPMM_PROGRAM_ID.to_string(), // Listen to raydium_cpmm program ID
RAYDIUM_CLMM_PROGRAM_ID.to_string(), // Listen to raydium_clmm program ID
PUMPFUN_PROGRAM_ID.to_string(), // Monitor pumpfun program ID
PUMPSWAP_PROGRAM_ID.to_string(), // Monitor pumpswap program ID
BONK_PROGRAM_ID.to_string(), // Monitor bonk program ID
RAYDIUM_CPMM_PROGRAM_ID.to_string(), // Monitor raydium_cpmm program ID
RAYDIUM_CLMM_PROGRAM_ID.to_string(), // Monitor raydium_clmm program ID
"xxxxxxxx".to_string(), // Monitor xxxxx account
];
let account_exclude = vec![];
let account_required = vec![];
@@ -141,11 +150,13 @@ async fn test_grpc() -> Result<(), Box<dyn std::error::Error>> {
async fn test_shreds() -> Result<(), Box<dyn std::error::Error>> {
println!("Subscribing to ShredStream events...");
// Create ShredStream client with performance monitoring enabled
let shred_stream = ShredStreamGrpc::new_with_config(
"http://127.0.0.1:10800".to_string(),
true, // enable performance monitoring
).await?;
// Create low-latency configuration
let mut config = ShredClientConfig::low_latency();
// Enable performance monitoring, has performance overhead, disabled by default
config.enable_metrics = true;
let shred_stream =
ShredStreamGrpc::new_with_config("http://127.0.0.1:10800".to_string(), config).await?;
let callback = create_event_callback();
let protocols = vec![
Protocol::PumpFun,
@@ -275,136 +286,6 @@ src/
└── main.rs # Example program
```
## Performance Optimizations
### Recent Performance Improvements
The latest version includes significant performance optimizations that dramatically improve event processing throughput:
#### 1. **Batch Processing System**
- **Event Batching**: Events are now processed in batches (default: 100 events per batch) instead of individually
- **Reduced Callback Overhead**: Batch processing reduces the number of callback invocations by up to 100x
- **Improved Throughput**: Significantly higher event processing rates with lower CPU usage
- **Configurable Batch Size**: Adjustable batch size to balance latency vs throughput
#### 2. **Memory Optimization**
- **Object Pooling**: `EventMetadataPool` and `TransferDataPool` reduce memory allocations
- **Pre-allocated Vectors**: `Vec::with_capacity()` for collections to avoid dynamic resizing
- **Reduced Cloning**: Minimized unnecessary data cloning operations
- **Memory Usage Monitoring**: Real-time memory usage tracking in performance metrics
#### 3. **Caching System**
- **Event Parse Cache**: `EventParseCache` avoids redundant transaction parsing
- **Cache Hit Rate Monitoring**: Track cache effectiveness in performance metrics
- **Intelligent Cache Management**: Automatic cache size management
#### 4. **Performance Monitoring**
- **Real-time Metrics**: Built-in performance monitoring with automatic display
- **Comprehensive Statistics**: Events/second, processing times, memory usage, cache hit rates
- **Configurable Monitoring**: Enable/disable performance monitoring as needed
- **Zero Overhead**: Monitoring can be completely disabled for maximum performance
#### 5. **Concurrent Processing**
- **Async Event Processing**: Non-blocking event handling with `tokio`
- **Parallel Protocol Parsing**: Multiple protocols parsed concurrently
- **Optimized Channel Sizes**: Increased channel capacity (5000) to handle high event volumes
### Performance Metrics
The built-in performance monitoring provides detailed insights:
- **Events Processed**: Total number of events processed
- **Events/Second**: Real-time processing rate (5-second rolling window)
- **Average Processing Time**: Mean time to process events
- **Min/Max Processing Time**: Fastest and slowest processing times
- **Cache Hit Rate**: Percentage of cache hits for event parsing
- **Memory Usage**: Estimated memory consumption
### Configuration Options
```rust
// Performance monitoring configuration
let grpc = YellowstoneGrpc::new_with_config(
endpoint,
x_token,
true, // enable performance monitoring
)?;
// Batch processing is automatically enabled with optimal settings
// Batch size: 100 events
// Batch timeout: 10ms
// Channel size: 5000
```
## Performance Considerations
1. **Connection Management**: Properly handle connection lifecycle and reconnection
2. **Event Filtering**: Use protocol filtering to reduce unnecessary event processing
3. **Memory Management**: Implement proper cleanup for long-running streams
4. **Error Handling**: Robust error handling for network issues and service disruptions
5. **Batch Processing**: Leverage batch processing for high-throughput scenarios
6. **Performance Monitoring**: Use built-in metrics to optimize your application
## Configuration Options
### Yellowstone gRPC Configuration
```rust
// Recommended: Create gRPC client with performance monitoring enabled
let grpc = YellowstoneGrpc::new_with_config(
"https://solana-yellowstone-grpc.publicnode.com:443".to_string(),
None,
true, // enable performance monitoring
)?;
// Alternative: Basic configuration (performance monitoring enabled by default)
let grpc = YellowstoneGrpc::new(
"https://solana-yellowstone-grpc.publicnode.com:443".to_string(),
None,
)?;
// Maximum performance: Disable performance monitoring
let grpc = YellowstoneGrpc::new_with_config(
"https://solana-yellowstone-grpc.publicnode.com:443".to_string(),
None,
false, // disable performance monitoring
)?;
```
### ShredStream Configuration
```rust
// Recommended: Create ShredStream client with performance monitoring enabled
let shred_stream = ShredStreamGrpc::new_with_config(
"http://127.0.0.1:10800".to_string(),
true, // enable performance monitoring
).await?;
// Alternative: Basic configuration (performance monitoring enabled by default)
let shred_stream = ShredStreamGrpc::new("http://127.0.0.1:10800".to_string()).await?;
// Maximum performance: Disable performance monitoring
let shred_stream = ShredStreamGrpc::new_with_config(
"http://127.0.0.1:10800".to_string(),
false, // disable performance monitoring
).await?;
```
### Performance Tuning
```rust
// Runtime performance monitoring control
grpc.set_enable_metrics(true).await; // Enable monitoring
grpc.set_enable_metrics(false).await; // Disable monitoring
// Get current performance metrics
let metrics = grpc.get_metrics().await;
println!("Current performance: {:?}", metrics);
// Manual performance metrics display
grpc.print_metrics().await;
```
## License
MIT License
@@ -414,6 +295,15 @@ MIT License
- Project Repository: https://github.com/0xfnzero/solana-streamer
- Telegram Group: https://t.me/fnzero_group
## Performance Considerations
1. **Connection Management**: Properly handle connection lifecycle and reconnection
2. **Event Filtering**: Use protocol filtering to reduce unnecessary event processing
3. **Memory Management**: Implement appropriate cleanup for long-running streams
4. **Error Handling**: Robust error handling for network issues and service interruptions
5. **Batch Processing Optimization**: Use batch processing to reduce callback overhead and improve throughput
6. **Performance Monitoring**: Enable performance monitoring to identify bottlenecks and optimization opportunities
## Important Notes
1. **Network Stability**: Ensure stable network connection for continuous event streaming