CogVLM2
cogvlm2-video
CogVLM2 is a model that combines image and video understanding, enabling tasks like captioning, visual question answering, and multimodal analysis.
Model Information
Input
Configure model parameters
Output
View generated results
Result
Preview, share or download your results with a single click.
Prerequisites
- Create an API Key from the Eachlabs Console
- Install the required dependencies for your chosen language (e.g., requests for Python)
API Integration Steps
1. Create a Prediction
Send a POST request to create a new prediction. This will return a prediction ID that you'll use to check the result. The request should include your model inputs and API key.
import requestsimport timeAPI_KEY = "YOUR_API_KEY" # Replace with your API keyHEADERS = {"X-API-Key": API_KEY,"Content-Type": "application/json"}def create_prediction():response = requests.post("https://api.eachlabs.ai/v1/prediction/",headers=HEADERS,json={"model": "cogvlm2-video","version": "0.0.1","input": {"top_p": 0.1,"prompt": "Describe this video.","input_video": "your input video here","temperature": 0.1,"max_new_tokens": 2048}})prediction = response.json()if prediction["status"] != "success":raise Exception(f"Prediction failed: {prediction}")return prediction["predictionID"]
2. Get Prediction Result
Poll the prediction endpoint with the prediction ID until the result is ready. The API uses long-polling, so you'll need to repeatedly check until you receive a success status.
def get_prediction(prediction_id):while True:result = requests.get(f"https://api.eachlabs.ai/v1/prediction/{prediction_id}",headers=HEADERS).json()if result["status"] == "success":return resultelif result["status"] == "error":raise Exception(f"Prediction failed: {result}")time.sleep(1) # Wait before polling again
3. Complete Example
Here's a complete example that puts it all together, including error handling and result processing. This shows how to create a prediction and wait for the result in a production environment.
try:# Create predictionprediction_id = create_prediction()print(f"Prediction created: {prediction_id}")# Get resultresult = get_prediction(prediction_id)print(f"Output URL: {result['output']}")print(f"Processing time: {result['metrics']['predict_time']}s")except Exception as e:print(f"Error: {e}")
Additional Information
- The API uses a two-step process: create prediction and poll for results
- Response time: ~12 seconds
- Rate limit: 60 requests/minute
- Concurrent requests: 10 maximum
- Use long-polling to check prediction status until completion
Overview
CogVLM2-Video is an advanced visual language model designed for comprehensive image and video understanding tasks. Building upon the previous generation, it offers significant improvements in various benchmarks, supports longer content lengths, higher image resolutions.
Technical Specifications
Base Model: Built upon Meta's Llama 3 with 8 billion parameters.
Multimodal Input: Capable of processing both textual and visual data, including images and videos.
Key Considerations
Performance: While the model achieves state-of-the-art results in many benchmarks, real-world performance may vary based on input quality and complexity.
Tips & Tricks
Effective Input Preparation:
- Text Prompts: Clearly articulate your prompts to guide the model effectively.
- Visual Inputs: Use high-quality images or videos within the supported resolution and length to enhance output accuracy.
Combining Modalities: Leverage both text and visual inputs simultaneously to enrich the context and improve the model's understanding.
Input Length: The model supports content lengths up to 8,000 tokens. Ensure your inputs stay within this limit to maintain optimal performance.
Image Resolution: For image inputs, resolutions up to 1344×1344 pixels are supported. Providing images within this resolution range will yield the best results.
Language Support: CogVLM2-Video is proficient in both Chinese and English. You can input prompts in either language based on your requirements.
Capabilities
Image Understanding: Analyzes and interprets high-resolution images, providing detailed insights and descriptions.
Video Understanding: Processes videos by analyzing keyframes, enabling comprehension of dynamic visual content.
What can I use for?
Visual Question Answering: Obtain answers to questions based on video content.
Content Analysis: Analyze visual media to extract meaningful information and summaries.
Things to be aware of
Interactive Applications: Create chatbots or virtual assistants that can interpret and respond to video input.
Educational Tools: Develop frameworks that use the model’s capabilities to provide explanations or summaries of video content for learning purposes.
Content Creation: Use the model to create descriptive content or narratives based on videos to assist with creative projects.
Limitations
Video Length: The model can process videos up to 1 minute in duration. Longer videos need to be truncated or segmented appropriately.
Resolution Constraints: Images exceeding 1344×1344 pixels may require downscaling to fit within the supported resolution.
Language Limitations: While proficient in Chinese and English, performance in other languages may be limited or unsupported.
Output Format: Text
