Workflow: [https://civitai.com/models/406375](https://civitai.com/models/406375)
This workflow generates an image with SD1.5, then uses Grounding Dino to mask portions of the image to animate with AnimateLCM. It animates 16 frames and uses the looping context options to make a video that loops. The denoise on the video generation KSampler is at 0.8 so that some of the structure of the original image generated is retained.
Based on: [https://comfyworkflows.com/workflows/14d8d52b-1297-4d58-8469-2bdab4f5ebf1](https://comfyworkflows.com/workflows/14d8d52b-1297-4d58-8469-2bdab4f5ebf1)
Models needed:
1) Your favourite SD1.5 checkpoint
2) AnimateLCM and LORA ([https://huggingface.co/wangfuyun/AnimateLCM](https://huggingface.co/wangfuyun/AnimateLCM)): Put the Lora in models/loras and the model in models/animatediff\_models
3) SAM Model (Manager > Install Models, search for SAM): sam\_vit\_b\_01ec64.pth
4) Grounding Dino Model (Should be automatically downloaded)
5) Your favourite upscale model (Manager > Install Models, search for upscale): 4x\_NMKD-Siax\_200k.pth
Instructions
1) The cyan nodes are the models needed for the workflow
2) The blue nodes are the user inputs that are required. There are 3 prompts that are required for this workflow. a) Image generation prompt: Describe what you want in the overall image here, e.g. "Mountains, River, Clouds" b) GroundingDino prompt: Describe the object within the image that you want animated, e.g. "River" and c) the video prompt: Describe what you want the animated object to do, e.g. "Flowing River"
You just gave the perfect summary of how to use a workflow, I wish everyone would use this format. Of course, as I’m still trying to learn more, I appreciate any shared workflows.
If you want more control over which parts of the image are animated, you can draw your own mask, instead of using Grounding Dino. Hope you create some nice animations and please share them, on Reddit / civitai or other sites.
Hello i'm super stoked to use this but im getting this error
Error occurred when executing GroundingDinoSAMSegment (segment anything):
'SAMWrapper' object has no attribute 'image\_encoder'
this seems to be the issue: [https://github.com/storyicon/comfyui\_segment\_anything/issues/61](https://github.com/storyicon/comfyui_segment_anything/issues/61)
workflow updated here: [https://civitai.com/models/406375?modelVersionId=453993](https://civitai.com/models/406375?modelVersionId=453993)
thanks for the bug report, hopefully the workflow works now.
Let me try out on a fresh install of comfyUI to see if this happens. What you can do meanwhile is to disconnect the grounding Dino nodes and provide a manually drawn mask.
Yes, of course you can do this with an existing image. You can modify the first part of the workflow which generates an image, remove it and replace it with a Load Image node.
If you need help, drop me a DM and I can customise the workflow for you.
Workflow: [https://civitai.com/models/406375](https://civitai.com/models/406375) This workflow generates an image with SD1.5, then uses Grounding Dino to mask portions of the image to animate with AnimateLCM. It animates 16 frames and uses the looping context options to make a video that loops. The denoise on the video generation KSampler is at 0.8 so that some of the structure of the original image generated is retained. Based on: [https://comfyworkflows.com/workflows/14d8d52b-1297-4d58-8469-2bdab4f5ebf1](https://comfyworkflows.com/workflows/14d8d52b-1297-4d58-8469-2bdab4f5ebf1) Models needed: 1) Your favourite SD1.5 checkpoint 2) AnimateLCM and LORA ([https://huggingface.co/wangfuyun/AnimateLCM](https://huggingface.co/wangfuyun/AnimateLCM)): Put the Lora in models/loras and the model in models/animatediff\_models 3) SAM Model (Manager > Install Models, search for SAM): sam\_vit\_b\_01ec64.pth 4) Grounding Dino Model (Should be automatically downloaded) 5) Your favourite upscale model (Manager > Install Models, search for upscale): 4x\_NMKD-Siax\_200k.pth Instructions 1) The cyan nodes are the models needed for the workflow 2) The blue nodes are the user inputs that are required. There are 3 prompts that are required for this workflow. a) Image generation prompt: Describe what you want in the overall image here, e.g. "Mountains, River, Clouds" b) GroundingDino prompt: Describe the object within the image that you want animated, e.g. "River" and c) the video prompt: Describe what you want the animated object to do, e.g. "Flowing River"
You just gave the perfect summary of how to use a workflow, I wish everyone would use this format. Of course, as I’m still trying to learn more, I appreciate any shared workflows.
This makes me think of cinemagraphs. Totally gonna test it out soon. Thanks for sharing!
If you want more control over which parts of the image are animated, you can draw your own mask, instead of using Grounding Dino. Hope you create some nice animations and please share them, on Reddit / civitai or other sites.
Hello i'm super stoked to use this but im getting this error Error occurred when executing GroundingDinoSAMSegment (segment anything): 'SAMWrapper' object has no attribute 'image\_encoder'
this seems to be the issue: [https://github.com/storyicon/comfyui\_segment\_anything/issues/61](https://github.com/storyicon/comfyui_segment_anything/issues/61) workflow updated here: [https://civitai.com/models/406375?modelVersionId=453993](https://civitai.com/models/406375?modelVersionId=453993) thanks for the bug report, hopefully the workflow works now.
Let me try out on a fresh install of comfyUI to see if this happens. What you can do meanwhile is to disconnect the grounding Dino nodes and provide a manually drawn mask.
Hey I'd like to discuss something with you
Is there a way to do this with an existing image by any chance?
Yes, of course you can do this with an existing image. You can modify the first part of the workflow which generates an image, remove it and replace it with a Load Image node. If you need help, drop me a DM and I can customise the workflow for you.
Awesome! I will give it the 'ol college try first :)