Auto "Alpha mask"

[pbacot]

Hi. Wonder if there's any chance of having the render not show the background (spherical, physial sky etc.) used for lighting-- someday. Is this a difficult coding issue? I use the alpha mask all the time, and it creates additional steps, but mostly, there is always some error in the edge cleanup when using it to remove the background, even at higher resolutions, which requires messing with borders and further cleanup. Thank you.

[Fletch]

Unfortunately there is no better mask feature than the current alpha mask pass.
Rendering the alpha mask at double resolution of the final rendered image and downsizing the mask in Photoshop will generally result in very good mask quality.

If it's not working for you, please post a link to some example files and we can see if we have any better mask advice.

[tuna57]

Going to "piggy back" on this post of pbacot's . Have the same issues when producing alpha mask to post pro in skies, grass and such. Posting an image of a mask I did last fall. Screenshot of the image opened in Gimp and zoomed in to see the problem. You end up with these various shades of grey running around everything in the mask image. Leaves a "halo" around all the objects and takes a massive amount of time to fix ( at least for me ). F.Y.I. the posted example was rendered at twice the size then scaled down for overlay.... Was going to post about this and then just kind of slipped under the rug. I sort of think it might have more to do with SketchUps not so good anti-ailiasing on Mac computers ?

[tuna57]

Here's a screenshot showing the alpha mask from my previous post after it was used to combine the render with a panoramic background image.....

[pbacot]

I wouldn't think it'd have to do with SketchUp's anti-aliasing, which it seems is just a screen rendering issue??? I thought Twilight handled AA separately. I once had a problem because I left post-pro settings in Twilight on for the alpha mask image--so there were shades instead of solid white. Otherwise I haven't noticed shades of gray, but I am concerned mostly with clean lines, not palm trees, which must be difficult. I'll get a fine white ghost edge on a roof line that is supposed to be dark. I can fix it but... every time gets to be a lot of busy work.

I will try doubling the Alpha Mask resolution though I hope that doesn't mean I have to try manually align the images.

Thank you! Peter

[SpookyChick]

Ouch! That looks like it was a pain to clean up,

I too have this problem at times. I also use GIMP for my post pro work, although I'm on a Win7x64pro machine. SketchUp's scummy antialising sometimes is a contributing factor to this problem. However, the primary culprit seems to be GIMP.

GIMP has a nasty habit of antialiasing the selection during the selection process, especially when selecting by color (Fuzzy Select as well). There should be controls in the toolbox / tool options menu to turn this off, depending on which version of GIMP you're using. However, as I have found, while that cures most of the problem, it doesn't fix all of it.

I will add my wish for the same thing; namely, a way to render using an HDRI or other spherical background for illumination, but have the HDR itself not appear in the final render. I suspect though, (and Chris or Fletch will probably tell us if this is the case) the Kerkythea engine doesn't support such a thing.

If there is a way to optionally turn background visibility in the final render on or off, or if it were possible to code in such an option, I would be overjoyed to find out about it.

Best of luck with finding a workaround!

Ja ne!

[tuna57]

pbacot, SpookyChick , I was just shooting in the dark at what may cause the problem. Was doing a lot of alpha masks last fall for practice and this was an ongoing problem. The weird shades of gray line between the white and black never takes on transparency when your your making your overlays , be it backgrounds or grass and such. At normal image size it just looks like a white ghost line around everything. Zoom way in and you see the gradients in that line.... I really don't think it's a Gimp issue ( could be wrong ) as I've seen the same stuff when working in Affinity. Hoping Fletch and Chris can shed some light.

[Fletch]

Here's a screenshot showing the alpha mask from my previous post after it was used to combine the render with a panoramic background image.....

Thanks for adding, Tuna - please, do you still have the pshop file you can send for analysis here at the lab?

I wouldn't think it'd have to do with SketchUp's anti-aliasing, which it seems is just a screen rendering issue??? I thought Twilight handled AA separately.

You are correct - so I am confused as to why one would mention SketchUp's AA in this instance - unless someone is exporting images for their masks using Sketchup's export image function? If so, then there's the problem.

I once had a problem because I left post-pro settings in Twilight on for the alpha mask image--so there were shades instead of solid white. Otherwise I haven't noticed shades of gray, but I am concerned mostly with clean lines, not palm trees, which must be difficult. I'll get a fine white ghost edge on a roof line that is supposed to be dark. I can fix it but... every time gets to be a lot of busy work.

I will try doubling the Alpha Mask resolution though I hope that doesn't mean I have to try manually align the images.

Thank you! Peter

Yes, Peter, if you are not rendering the Alpha Mask at double resolution, and downsize in your post-pro image editing software, then you will get ghosting. If you do it right, there will be little ghosting. It is a tried and true method.

That said, in the palm tree example, I am not sure what I'm looking at as everything seems so pixelated. I can say that masking a palm tree that has been photographed (or rendered) on a light background and then placing said tree onto a darker sky will not look good. This is because the fine fronds will needs-must be blended by the engine (either of the digital camera or of the render engine) with the surrounding pixels in order to get a smooth looking image. This is why I'm asking for the layered pshop file, as it's not clear what's going on in that image, but my suspicion is that one is trying to place a light-colored palm on a dark-colored background. That won't work in any render engine or any masking circumstance. The replacement sky should match the original sky in basic light/dark color range for a realistic masking result.

[tuna57]

Fletch , thanks for getting back to us about this. I stink at explaining things a lot of times ... so going to try and clarify some of it. You can ignore my comment about SU's anti aliasing as I was grasping at straws with that comment. I think the problem is in the alpha mask, between the black and the white there is a line of pixels that are neither black nor white. I'll try and describe my process , who knows maybe I'm doing things wrong. I open the alpha mask in my photo editor, select the black and clear it. Now I have transparent in the sky area and white where I want my render to show. Next I bring in my background image and get that arranged and merge the two images. I select the white area and clear that. Now I have my mask for over my render. Finally bring in the render below the mask and I have my image with the new background/sky. The images I previously posted were pixelated from zooming in so you could see the weird line that runs between the black and white in the alpha mask. Posting a final image of a test I ran today and the mask before clearing out the white. The line can be seen clearly. The more complex the mask the crazier it looks ( the palm trees ). If you'd still like a psd file of one of these things it will have to wait a few days until I get my desktop machine out of the shop and back up and running.

[Fletch]

I think the problem is in the alpha mask, between the black and the white there is a line of pixels that are neither black nor white.

This is essential to any mask - it's the anti-aliasing that will blend the edge of the foreground with the background.

..., who knows maybe I'm doing things wrong. I open the alpha mask in my photo editor, select the black and clear it.

OH, THIS IS THE PROBLEM. ...no no no... This is the reason for the ghosting and for the pixelated edge.
There is no need to select and delete any part of the alpha mask image in your photo editor. Simply use the Alpha Mask Rendering as the mask for your layer.
Please follow this tutorial carefully:
Subject: Background Image / Picture Using Alpha Mask

If you'd still like a psd file of one of these things ...

No need, thanks for the clear explanation - please see these threads:
Subject: Background Image / Picture Using Alpha Mask
Subject: How to use Alpha Mask render pass in Post Production
Subject: Force background image to render unaffected by lighting